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Abstract 


The  purpose  of  the  COMmand  Decision  Aiding  Technology  (COMDAT)  Technology 
Demonstrator  Project  (TDP)  was  to  research  and  demonstrate  multi-source  data  fusion  (MSDF) 
technologies  and  carry  out  human  factors  (HF)  studies  to  support  upgrades  to  the  Halifax  Class 
Command  and  Control  System  (CCS),  in  the  area  of  battle  space  awareness.  Since  the  requisite 
FIF  knowledge  to  support  this  TDP  did  not  exist,  the  F1F  component  included  the  generation  of  the 
founding  knowledge  necessary  for  decision  aid  development  in  future  projects.  The  major  work 
items  included 

•  the  conduct  of  cognitive  and  functional  task  analyses  to  define  the  information 
requirements  of  the  major  Operations  Room  positions  on  Flalifax  Class  ships; 

•  the  development  of  human- in-the-loop  measures  of  performance  in  order  to  evaluate 
strategies  for  conducting  performance  evaluations,  define  current  performance  in  the 
Operations  Room,  and  evaluate  technologies  including  MSDF; 

•  recommendations  for  improvements  to  human  computer  interfaces  in  the  existing  CCS 
and  the  Technology  Demonstrator  (TD); 

•  the  development  of  a  concept  of  operations  for  user  involvement  in  MSDF;  and 

•  support  for  an  evaluation  of  operator-system  picture  compilation  performance  using  the 
final  TD. 

The  detailed  results  of  the  FIF  work  have  been  published  in  a  series  of  reports  over  the  course  of 
the  project  which  lasted  from  June  2000  to  March  2007.  This  final  report  provides  a  summary  of 
the  reports  along  with  related  material  that  informed  and  supplemented  this  work,  some  lessons 
learned,  and  recommendations  for  future  work.  Overall,  we  have  a  better  understanding  of  the 
usefulness  of  the  methods  that  were  investigated  as  well  as  considerably  more  knowledge  about 
the  information  requirements  and  processes  involved  in  building  the  tactical  picture  in  the  Flalifax 
Class  Operations  Room.  In  addition,  we  have  generated  improved  guidance  for  designing  the 
interfaces  needed  to  support  operator  goals  and  tasks  as  well  as  a  better  understanding  of  the 
research  needed  to  build  even  more  effective  interfaces  and  decision  support  systems  for  naval 
command  and  control. 
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Resume 


Le  projet  de  demonstration  de  la  technologie  (PDT)  d'aide  aux  decisions  de  commandement 
(COMDAT)  avait  pour  objet  d’etudier  les  technologies  de  fusion  de  donnees  multi-sources 
(MSDF)  et  d’en  faire  la  demonstration  ainsi  que  de  mener  des  etudes  des  facteurs  humains  (FH)  a 
l’appui  de  mises  a  niveau  du  systeme  de  commandement  et  de  controle  (C2)  des  navires  de  la 
classe  Flalifax,  dans  le  domaine  de  la  connaissance  de  l’espace  de  combat.  Com  me  la  connaissance 
des  FF1  necessaire  pour  appuyer  ce  PDT  n’existait  pas,  la  composante  FF1  comprenait  la  generation 
des  connaissances  de  base  necessaires  a  l’elaboration  d’aides  aux  decisions  dans  des  projets  futurs. 
Les  principales  taches  de  recherche  comprenaient : 

•  l’execution  d’analyses  des  taches  cognitives  et  fonctionnelles  pour  definir  les  exigences 
d’ information  des  postes  principaux  dans  la  salle  des  operations  des  navires  de  la  classe 
Flalifax; 

•  F  elaboration  de  mesures  de  rendement  avec  un  chainon  humain  dans  la  boucle  afm 
d’evaluer  des  strategies  devaluation  du  rendement,  de  definir  le  rendement  actuel  dans  la 
salle  des  operations  et  d’evaluer  les  technologies,  y  compris  la  MSDF; 

•  des  recommandations  d’ amelioration  des  interfaces  humain-ordinateur  dans  le  systeme  C2 
existant  et  le  demonstrateur  de  la  technologie  (DT); 

•  Felaboration  d’un  concept  des  operations  en  vue  de  la  contribution  des  utilisateurs  a  la 
MSDF; 

•  le  soutien  d’une  evaluation  du  rendement  de  compilation  d’un  tableau  de  l’interface 
operateur-systeme  au  moyen  du  DT  definitif. 

Les  resultats  detailles  des  travaux  en  FH  ont  ete  publies  dans  une  serie  de  rapports  tout  au  long  du 
projet,  qui  a  dure  de  juin  2000  a  mars  2007.  Le  present  rapport  final  resume  ces  rapports  de  meme 
que  du  materiel  connexe  qui  soutenait  et  completait  ces  travaux,  certaines  logons  apprises  et  des 
recommandations  de  recherches  futures.  Somme  toute,  nous  avons  une  meilleure  comprehension 
de  l’utilite  des  methodes  etudiees,  ainsi  qu’une  connaissance  beaucoup  plus  complete  de 
F information  requise  et  des  processus  necessaires  pour  constituer  le  tableau  tactique  dans  la  salle 
des  operations  des  navires  de  la  classe  Halifax.  De  plus,  nous  avons  elabore  des  directives 
ameliorees  pour  concevoir  les  interfaces  necessaires  pour  appuyer  les  objectifs  et  taches  des 
operateurs  et  acquis  une  meilleure  comprehension  des  recherches  necessaires  pour  construire  des 
interfaces  et  des  systemes  d’aide  aux  decisions  encore  plus  efficaces  pour  le  commandement  et  le 
controle  navals. 
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Executive  Summary 


Introduction: 

Advances  in  threat  technology,  the  increasing  tempo  and  diversity  of  littoral  scenarios,  and  the 
volume  and  ambiguous  nature  of  data  to  be  processed  under  time-critical  conditions  pose 
significant  challenges  for  Command  and  Control  Systems  (CCS)  in  Canadian  ships  as  well  as  the 
operators  who  must  use  these  systems  to  defend  their  ship  and  fulfil  their  mission.  Consequently, 
since  the  early  1990’s  the  Canadian  Navy  has  assigned  a  high  priority  to  research  and 
development  leading  to  enhanced  command,  control,  and  information  system  capabilities  in  data 
and  information  fusion  and  decision  support  to  provide  an  accurate,  and  more  timely,  integrated 
Maritime  Tactical  Picture  in  order  to  provide  the  Command  Team  with  as  complete  an 
understanding  of  the  situation  and  threats  as  possible.  The  COMmand  Decision  Aiding 
Technology  (COMDAT)  Technology  Demonstrator  Project  (TDP)  was  initiated  to  support  that 
effort  through  research  and  development  in  multi-source  data  fusion  (MSDF)  technologies  and 
related  human  factors  (HF)  studies.  The  purpose  of  the  current  report  is  to  summarize  the  F1F  work 
carried  out  under  COMDAT.  Since  the  requisite  FIF  knowledge  to  support  this  TDP  did  not  exist, 
the  HF  component  included  the  generation  of  the  founding  knowledge  necessary  for  decision  aid 
development  in  future  projects. 

Results: 

This  final  report  provides  a  summary  of  the  HF  work  carried  out  under  COMDAT  along  with 
related  material  that  informed  and  supplemented  this  work,  some  lessons  learned  and 
recommendations  for  future  work.  The  major  HF  outcomes  of  the  TDP  were  improved  methods, 
tools,  and  knowledge  about 

•  cognitive  and  functional  task  analyses  to  define  the  information  requirements  of  the  major 
Operations  Room  positions  on  Halifax  Class  ships, 

•  the  development  of  human- in-the-loop  measures  of  performance  to  define  the  current 
performance  in  the  Operations  Room  and  evaluate  technology  including  MSDF, 

•  the  design  of  human  computer  interfaces  in  naval  command  and  control  systems, 

•  the  design  of  a  concept  of  operations  for  user  involvement  in  MSDF,  and 

•  the  design  and  conduct  of  the  evaluation  of  operator  performance  on  existing  and  new 
technologies. 

Significance: 

The  results  of  this  work  have  already  informed  the  mid-life  upgrade  of  the  Halifax  Class 
Operations  Room  and  are  expected  to  define  the  HF  involvement  in  future  naval  acquisition 
projects. 

Future  plans: 

Some  of  the  recommendations  for  further  work  are  being  undertaken  in  other  TDPs  such  as 
INCOMMANDS  and  in  naval  applied  research  projects. 
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Sommaire 


Introduction: 

Les  progres  de  la  technologie  de  la  menace,  la  cadence  acceleree  et  la  diversite  croissante  des 
scenarios  sur  les  littoraux,  ainsi  que  le  volume  et  la  nature  ambigue  des  donnees  a  traiter  dans  des 
delais  critiques  posent  d'importants  defis  aux  systemes  de  commandement  et  de  controle  (C2)  a 
bord  de  navires  canadiens,  ainsi  qu’aux  operateurs  qui  doivent  utiliser  ces  systemes  pour  defendre 
leur  navire  et  s'acquitter  de  leur  mission.  Par  consequent,  depuis  le  debut  des  annees  1990,  la 
Marine  canadienne  a  accorde  une  haute  priorite  a  la  recherche  et  developpement  menant  a  des 
capacites  ameliorees  du  systeme  de  commandement,  de  controle  et  d’information  a  l’egard  de  la 
fusion  des  donnees  et  de  l’information  et  de  l’aide  aux  decisions  afm  de  foumir  un  tableau  tactique 
maritime  integre  exact  et  plus  opportun  dans  le  but  de  foumir  a  l’equipe  de  commandement  une 
comprehension  de  la  situation  et  des  menaces  aussi  complete  que  possible.  Le  projet  de 
demonstration  de  la  technologie  (PDT)  d'aide  aux  decisions  de  commandement  (COMDAT)  a  ete 
lance  pour  appuyer  cette  initiative  grace  a  la  recherche  et  developpement  en  technologie  de  fusion 
de  donnees  multi-sources  (MSDF)  et  a  des  etudes  connexes  des  facteurs  humains  (FH).  Le  present 
rapport  a  pour  but  de  resumer  les  travaux  en  FH  effectues  dans  le  cadre  de  la  COMDAT.  Comme 
la  connaissance  des  FH  necessaire  pour  appuyer  ce  PDT  n’existait  pas,  la  composante  FH 
comprenait  la  generation  des  connaissances  de  base  necessaires  a  l’elaboration  d’aides  aux 
decisions  dans  des  projets  futurs. 

Resultats: 

Le  present  rapport  final  resume  les  travaux  en  FH  effectues  dans  le  cadre  de  la  COMDAT  de 
meme  que  le  materiel  connexe  qui  soutenait  et  completait  ces  travaux,  certaines  lesons  apprises  et 
des  recommandations  de  recherches  futures.  Les  principaux  resultats  en  termes  de  FH  du  PDT 
etaient  des  methodes,  outils  et  connaissances  ameliores  dans  les  domaines  suivants  : 

•  analyses  des  taches  cognitives  et  fonctionnelles  pour  definir  les  exigences  d’information 
des  postes  principaux  dans  la  salle  des  operations  des  navires  de  la  classe  Halifax; 

•  elaboration  de  mesures  de  rendement  avec  un  chainon  humain  dans  la  boucle  afm  de 
definir  le  rendement  actuel  dans  la  salle  des  operations  et  d’evaluer  les  technologies,  y 
compris  la  MSDF; 

•  conception  d’interfaces  humain- ordinateur  dans  les  systemes  de  commandement  et  de 
controle  navals, 

•  elaboration  d’un  concept  des  operations  en  vue  de  la  contribution  des  utilisateurs  a  la 
MSDF; 

•  conception  et  execution  de  F evaluation  du  rendement  des  operateurs  utilisant  des 
technologies  existantes  et  nouvelles. 

Porte'e: 

Les  resultats  de  ces  travaux  ont  deja  contribue  a  la  formulation  de  la  mise  a  niveau  a  demi-vie  de 
la  salle  des  operations  des  navires  de  la  classe  Halifax,  et  Ton  s’attend  a  ce  qu’ils  definissent  la 
contribution  des  FH  aux  projets  d’acquisition  navals. 
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Recherches  futures: 

Certaines  des  recommandations  de  recherches  futures  sont  mises  en  oeuvre  dans  le  cadre  d’autres 
PDT  tels  qu’INCOMMANDS  et  de  projets  de  recherche  appliquee  navale. 
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HU  MAN. 9  Y  STEMS 


1  Background 


The  current  Halifax  Class  Command  and  Control  System  (CCS)  is  based  on  a  concept  of 
operations  and  technology  that  derive  from  the  late  70’s  and  early  80’s.  Advances  in  threat 
technology,  the  increasing  tempo  and  diversity  of  littoral  scenarios,  along  with  the  volume  and 
ambiguous  nature  of  data  to  be  processed  under  time-critical  conditions  pose  significant  challenges 
for  the  CCS  and  the  operators  who  use  these  systems  to  defend  their  ship  and  fulfil  their  mission. 
Consequently,  since  the  1990’s  the  Canadian  Navy  has  assigned  a  high  priority  to  research  and 
development  leading  to  enhanced  Command  and  Control  (C2)  capabilities  in  data  fusion  and 
decision  support  to  provide  an  accurate  integrated  Maritime  Tactical  Picture  (MTP)  in  order  to 
provide  the  Command  Team  with  as  complete  an  understanding  of  the  tactical  situation  as  possible. 

In  response  to  this  requirement,  Defence  Research  and  Development  Canada  (DRDC)  established 
the  Command  Decision  Support  Working  Group  (CDS  WG)  to  determine  the  Research  and 
Development  (R&D)  required  for  improved  decision  support  for  the  Command  Team.  Many  of  the 
recommendations  of  the  CDS  WG  (McCann,  Chalmers,  Flowers,  Hix,  Paradis,  Pigeau,  and  Turner 
1998)  centred  on  medium  and  long  term  research  in  human  factors  (HF)  including  Human- 
Computer  Interaction  (HCI)  and  method  and  measures  for  assessing  human-system  performance. 

In  addition,  there  was  a  requirement  for  behavioural  and  cognitive  analyses  to  support  the 
development  of  effective  decision  support.  They  recommended  the  following  R&D  activities: 

•  cognitive  task  analysis  of  the  Operations  Room  Officer  (ORO)  function, 

•  literature  survey  of  technologies  for  and  HF  aspects  of  naval  decision  support,  and 

•  Cognitive  Work  Analysis  (CWA)  and  Operator  Machine  Interface  (OMI)  requirements  for 
computer-based  decision  support  tools  for  the  ORO. 

In  response  to  this  recommendation,  DRDC  Toronto  carried  out  a  cognitive  task  analysis  of  the 
ORO  (Matthews,  Webb,  and  Bryant  1999b)  and  a  literature  survey  on  HF  issues  related  to  decision 
support  (Bryant  and  Webb  1999)  and  DRDC  Valcartier  conducted  a  proof  of  concept  study  on 
CWA  (Chalmers,  Bums,  and  Bryant  2001).  The  results  of  these  studies,  together  with  analyses  of 
current  deficiencies  in  the  Halifax  Class  Operations  Room,  led  to  recommendations  for  further 
research  and  design  concepts  for  improving  the  CCS. 

To  support  the  short  term  requirements  for  improved  decision  support,  DRDC  approved  the 
Command  Decision  Aiding  Technology  (COMDAT)  Technology  Demonstrator  Project  (TDP)  to 
transition  existing  research  on  Multi-Source  Data  Fusion  (MSDF)  into  the  Canadian  Forces.  Since 
the  corresponding  HF  knowledge  was  not  available,  COMDAT  included  a  large  HF  component  to 
provide  both  direct  support  to  the  development  of  the  Technology  Demonstrator  (TD)  and  the 
founding  knowledge  for  decision  aid  development  in  future  projects.  The  major  HF  work  items 
were  as  follows: 

•  Conduct  a  function  and  task  analysis  of  the  Halifax  Operations  Room  positions  to  more 
fully  understand  their  information  requirements  and  how  the  positions  interact  in  order  to 
evaluate  the  potential  impact  of  MSDF  and  to  define  the  HCI  requirements  for  future 
decision  support  systems. 

•  Define  Human-in-the-Loop  (HIL)  Measures  of  Performance  (MOPs)  and  trial  some 
selected  MOPs. 
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•  Evaluate  performance  on  the  existing  CCS  using  the  MOPs  developed  under  the  previous 
work  item. 

•  Suggest  improvements  to  the  OMI  of  the  existing  CCS  and  the  TD. 

•  Develop  Concepts  of  Operations  (CONOPS)  for  user  involvement  in  MSDF. 

•  Support  the  conduct  of  an  evaluation  of  operator  performance  on  the  final  TD. 

1 .1  Goals  of  the  HF  program  to  support  the  COMDAT  TD 

The  ultimate  goal  of  the  above  work  was  to  develop  a  body  of  HF  knowledge  that  could  be  used  to 
support  the  TD  as  well  as  provide  the  founding  knowledge  for  decision  aid  development  in  future 
projects.  Specific  goals  for  the  HF  work  included 

•  enhanced  knowledge  of  which  OMIs  are  feasible  and  operationally  useful,  and 

•  the  required  domain  knowledge  to  establish  the  evaluation  capabilities  to  conduct  HF  work 
in  future  TDPs. 

It  was  anticipated  that  this  information  would  be  provided  through  a  series  of  reports  produced 
during  the  TDP  including  the  current  report. 

1 .2  Structure  of  report 

Section  2  of  the  report  provides  a  summary  of  the  most  relevant  research  that  informed  the  HF 
work  under  COMDAT.  Prior  to  COMDAT’s  start,  DRDC  Toronto  was  actively  researching  the 
development  of  improved  HF  methods  and  measures  to  support  the  acquisition  of  new  systems  in 
the  Canadian  Forces.  In  addition  other  nations,  in  particular  the  US,  were  engaged  in  related  efforts 
to  develop  a  human  centered  approach  to  the  design  of  naval  systems.  DRDC  had  access  to  much 
of  this  work  through  its  participation  in  The  Technical  Cooperation  Programme  (TTCP),  Human 
Resources  and  Performance  Group  (HUM)  Technical  Panel  9  (TP9):  Human  Systems  Integration 
for  Naval  Systems.  A  brief  summary  of  the  most  relevant  aspects  of  that  work  is  also  included. 

Sections  3  through  8  summarize  the  work  completed  under  each  of  the  COMDAT  HF  work  units. 
Each  of  these  sections  has  the  following  structure: 

•  title  of  work  unit, 

•  goals  of  the  work, 

•  parties  responsible  for  the  work, 

•  list  of  reports  produced,  and 

•  overview  of  each  report  including 

abstract  (from  the  report), 
summary  of  work  conducted 

discussion  and  commentary  from  the  present  authors. 

The  report  concludes  with  a  section  which  outlines  some  lessons  learned  from  the  process  and 
identifies  areas  where  additional  research  may  be  required. 
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2  Prior  Work 


2.1  Relevant  human  factor  research  at  DRDC  Toronto 

2.1.1  A  framework  for  evaluating  military  command  and  control  systems 

Before  the  start  of  the  COMDAT  TDP,  DRDC  Toronto  was  actively  engaged  in  developing  generic 
requirements  and  methodologies  that  would  provide  guidelines  to  demonstrate  how  HF  issues 
could  be  systematically  integrated  within  a  R&D  program.  The  most  relevant  work  for  the 
COMDAT  TDP  was  the  specification  of  an  overall  framework  to  guide  the  process  of  evaluating 
performance  of  C2  systems  that  reflected  both  the  contributions  of  the  system  hardware  and 
software  as  well  as  human  operators. 

One  significant  publication  to  come  out  of  this  effort  was  the  report: 

“Matthews,  M.  L.,  Webb,  R.  D.  G.,  and  McCann,  C.  (1997).  A  framework  for  evaluation  of 
military  command  and  control  systems.  (DCIEM  No.  97-CR-24).  Wumansystems 
Incorporated,  Guelph,  ON.” 

A  major  conclusion  of  this  report  was  that  measures  of  performance  provide  more  diagnosticity 
when  they  focus  on  system  processes  than  outcome  effectiveness.  Further,  the  latter  class  of 
measures  are  often  difficult  to  obtain  under  the  required  operational  conditions  and  may  lack 
sufficient  reliability  and  validity,  because  of  lack  of  control  of  critical  variables  of  influence. 

Three  different  human-system  domains  were  identified  as  key  to  the  evaluation  of  system 
effectiveness: 

•  Function  Usability:  compliance  of  system  features,  the  Operator  Machine  Interface  (OMI)  and 
system  functionality  with  HF  guidelines  and  evaluated  largely  in  terms  of  ease  of  use  and 
utility; 

•  User  Fit:  demonstrated  ability  of  the  system  to  support  user  needs  in  key  areas  such  as 

-  situation  awareness, 

-  communication  effectiveness, 

-  decision  support, 

-  mental  workload, 

-  physical  layout  and  environmental  factors; 

•  Operational  Impact:  overall  ability  of  the  system  to  support  users,  working  under 
representative  conditions,  to  meet  mission  goals  and  sub-goals. 

To  provide  a  basis  for  the  systematic  capture  of  MOPs  for  critical  C2  tasks,  it  was  proposed  that 
standard  scenarios  be  created.  These  would  range  in  complexity  from  simple  strings  of  tasks  or 
sub-tasks  to  more  complex  mission  representations  in  order  to  challenge  the  capability  of  the 
system  to  meet  the  needs  of  users,  as  they  work  to  achieve  mission  goals  at  the  individual  or  team 
level.  Using  such  standard  scenarios,  the  three  human-system  domains  would  be  evaluated  using 
three  distinct  configurations  employing  simulation  fidelity  appropriate  to  the  stage  of  system 
development.  The  three  evaluation  configurations  are 
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•  Storys  Board :  Early  in  system  development,  operational  and  HF  Subject  Matter  Experts 
(SMEs)  systematically  would  walk  through  task  based  scenarios  using  low  fidelity  part  or 
whole  representations  of  the  system  concept  (mock  ups  or  story  boards).  These  SMEs 
would  assess  features  and  predicted  functionality  for  each  evaluation  domain  (including 
compliance  with  E1F  guidelines)  in  terms  of  projected  usability,  user  fit,  and  operational 
impact.  Concept  acceptance  would  be  based  on  the  outcome  of  this  evaluation. 

•  Test  bed:  This  evaluation  configuration  evolves  in  fidelity  over  the  course  of  system 
development: 

a.  Level  1 :  low  fidelity  representation  of  console,  screens,  or  floor  layout, 

b.  Level  2:  static  test  screens  representing  basic  system  part  task  functions, 

c.  Level  3:  part  system  functionality  with  task  strings  performed  interactively, 

d.  Level  4:  fully  functional  prototypes  configured  for  typical  team  operation(s). 

•  Field  trial  or  full  scale  simulation :  In  this  configuration,  the  full  system  is  evaluated  in  a 
typical  operational  context.  This  would  normally  occur  after  installation  at  the  first 
operational  site. 

The  methods  used  to  capture  MOP  data  would  vary  across  the  three  evaluation  configurations, 
although  some  methods,  such  as  rating  scales  and  questionnaires,  apply  over  all  three 
configurations.  At  the  story  board  level,  OMI  aspects  of  individual  system  components  would  be 
assessed  against  HF  guidelines  and  design  recommendations  and  checklists  and  ratings  completed 
by  HF  and  operational  SMEs  would  be  employed  to  assess  basic  elements  of  feature  usability.  In 
the  test  bed  and  field  trial  configurations,  measures  of  effectiveness  would  be  based  on  how  well 
and  how  quickly  mission  critical  tasks  and  goals  are  accomplished  using  prepared  scenarios  with 
the  most  typical  measures  being  task  completion  time  and  accuracy. 

As  part  of  the  literature  review  conducted  for  this  report,  several  other  important  issues  concerning 
the  application  of  HF  to  the  system  development  process  were  noted: 

•  Integrate  verification  ( Did  the  developers  do  the  right  things?)  and  validation  (Does  the 
system  perform  effectively  with  users  in  the  loop?). 

•  Apply  HF  from  the  start  of  development  including  concept  acceptance. 

•  Have  the  HF  effort  be  independent  of  the  development  group. 

•  Apply  scientific  reliability  and  rigor  consistently  in  the  measurement  process  to  ensure  that 
evaluation  results  will  predict  likely  operational  outcomes  with  some  degree  of  confidence. 

•  Account  for  the  context  of  operations:  for  example  work  volume  and  pattern, 
organizational  structure,  team  interaction,  training  of  test  participants,  workspace  layout, 
and  secondary  tasks  performed  in  parallel  with  the  primary  tasks  (e.g.  operation  of 
supplementary  systems,  supervision). 

The  report  also  provided  sample  MOPs  and  addressed  logistical  issues  concerning  the  capture  of 
performance  data  in  real  time. 

This  framework  was  used  as  the  basis  for  the  development  of  methods  and  measures  of 
performance  covered  in  Section  4  on  the  development  of  MOPS. 
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2.1 .2  Cognitive  task  analysis  of  the  Operations  Room  Officer  in  the  Halifax  Class 
frigate 

One  of  the  major  recommendations  of  the  CDS  WG  (McCann  et  al.  1998)  was  to  conduct  a 
Cognitive  Task  Analysis  (CTA)  of  the  ORO  position  in  the  Halifax  Class  frigate.  It  was  required  to 
better  understand  the  cognitive  aspects  of  C2  in  the  Operations  Room  (Ops  Room)  as  a  prelude  to 
preparing  evaluation  measures  to  assess  future  C2  support  systems.  The  ORO  position  was 
selected  for  analysis  from  among  other  Ops  Room  positions  as  being  the  position  with  the  most 
comprehensive  responsibilities,  most  likely  to  provide  a  broad  insight  into  C2  functions  and  to 
generalise  to  other  positions. 

The  reports  produced  under  this  work  package  were: 

Matthews,  M.  L.,  Webb,  R.  D.  G.,  and  Bryant,  D.  J.  (1999).  Cognitive  task  analysis  of  the  Halifax- 
class  Operations  Room  officer.  (DCIEM  No.  CR  1999-028).  Humansystems  Incorporated, 
Guelph,  ON. 

Matthews,  M.  L.,  Webb,  R.  D.  G.,  and  Bryant,  D.  J.  (1999).  Annexes  to:  Cognitive  task  analysis  of 
the  Halifax-class  Operations  Room  officer:  Data  sheets.  (DCIEM  No.  CR  1999-029). 
Humansystems  Incorporated,  Guelph,  ON. 

Matthews,  M.  L.  and  Webb,  R.  D.  G.  (2000).  Validation  of  cognitive  task  analysis  (CTA)  for  the 
Halifax-Class  ORO.  (DCIEM-CR-2000-066).  Humansystems  Incorporated,  Guelph,  ON. 

The  CTA  (Matthews,  Webb,  and  Bryant  1999a;  Matthews  et  al.  1999b)  was  based  on  interviews 
with  eight  pairs  of  experienced  OROs.  Each  interview  session  lasted  two  days  and  walked 
interviewees  through  an  operational  scenario,  which  was  based  on  a  training  exercise  from  the 
Halifax  Class  land  based  training  simulator.  OROs  were  prompted  to  describe  their  cognitive 
activities  based  on  experience  with  similar  events  to  those  in  the  scenario.  The  ORO  position  in 
question  was  in  a  Halifax  Class  vessel  in  the  forward  screen,  with  no  Task  Group  (TG)  warfare 
duties. 

Although,  conducted  in  the  context  of  one  position,  as  well  as  constrained  by  current  technology 
and  procedures,  the  objective  was  to  identify  functional  goals  and  associated  cognitive  needs  that 
were  device  independent.  Based  on  a  previous  review  of  the  literature  for  C2  systems,  four  key 
functional  areas  were  used  to  focus  the  interviews  and  for  development  of  evaluation  measures. 
These  were  situation  awareness,  communication,  decision  making,  and  mental  workload. 

Selected  observations  arising  from  the  interviews  (and  expanded  in  the  report)  include  the 
following: 

•  OROs  employ  a  variety  of  mental  models  to  organise  their  cognitive  demands.  Mental 
models  vary  in  terms  of  level  of  detail,  domain,  and  abstraction;  they  may  be  modulated  by 
career  path,  experience,  other  team  members,  mission  planning,  watch  hand-over,  and 
recent  events. 

•  Information  systems  need  to  provide  less  data  and  more  “information”  i.e.  pertinent  data 
meaningfully  integrated  in  terms  of  the  users’  mental  model(s).  This  implies  a  need  for 
functional  support  that  can  be  readily  adapted  by  “educated”  users  to  individual, 
contextual,  and  mission  related  needs. 

•  Mission  preparation  plays  a  major  role  in  establishing  mental  models  that  will  be  employed 
during  the  mission  operation. 
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•  The  core  ORO  function  is  to  manage  the  overall  Ops  Room  team  picture  compilation  and 
threat  response  activities,  rather  than  to  be  directly  involved  in  the  details  of  executing 
these  functions.  However,  the  ORO  does  become  involved  in  both  of  these  functions  when 
existing  team  resources  are  not  able  to  fully  manage  multiple  threats. 

•  In  the  case  of  situation  awareness,  major  requirements  include  information  acquisition  and 
integration,  cognitive  fit  of  the  available  data  to  the  mental  model(s)  of  the  user,  regaining 
awareness  after  switches  in  attention  between  different  areas  of  focus,  and  alerts  for 
unattended  areas. 

•  Background  tasks  such  as  handling  text  messages  interact  significantly  with  ongoing 
operational  responsibilities  and  require  the  ORO  frequently  to  switch  mental  models. 

•  Handling  of  multiple  threats  particularly  in  areas  of  high  traffic  density  are  likely  to  remain 
a  difficult  information  processing  challenge.  This  challenge  is  increased  if  the  ship  is 
assigned  TG  responsibility  for  a  warfare  area. 

After  the  work  was  finished,  there  was  some  concern  about  the  completeness  and  hence  overall 
accuracy  of  the  results  including 

•  the  completeness  and  accuracy  of  the  scenario  used  during  CTA  interviews, 

•  the  completeness  and  accuracy  of  goal  related  activities  identified  by  CTA  OROs, 

•  the  taxonomy  of  mental  models  used  by  OROs  to  manage  information,  and 

•  the  proposed  measures  of  cognitive  performance. 

A  validation  (Matthews  and  Webb  2000)  was  conducted  by  reviewing  the  CTA  scenario  and 
results  with  four  OROs  who  had  not  participated  in  the  original  CTA.  OROs  were  interviewed  in 
pairs  and  interviews  lasted  one  day  for  each  pair.  CTA  results  were  also  compared  with  standards 
for  evaluation  of  OROs  during  training  and  pertinent  elements  of  the  United  States  Universal  Naval 
Task  List. 

It  was  concluded  that  the  results  of  the  CTA  were  largely  accurate  and,  with  a  few  changes  in 
terminology,  could  probably  be  generalised  to  other  naval  missions  beyond  the  initial  scenario  used. 
Exceptions  or  areas  of  uncertainty  were  the  omission  of  certain  aspects  of  an  ORO’s  work  in  the 
CTA  scenario  rather  than  any  distortion  introduced  by  inaccuracies  in  the  scenario.  Further,  the 
original  CTA  scenario  was  found  to  have  insufficient  emphasis  on,  or  absence  of  the  following: 

•  an  overall  mission  objective  (as  opposed  to  reactions  to  individual  threat  events), 

•  ORO  planning  functions  before  and  during  a  mission, 

•  ORO  Task  Group  assignments  such  as  warfare  duties, 

•  ORO  co-ordination  or  de-confliction  of  ship  board  activities  from  a  mission  perspective,  and 

•  missions  such  as  humanitarian  aid  involving  interaction  with  organisations  outside  the 
Navy. 

The  structuring  of  the  results  of  the  CTA  into  background  (Ops  Room  management)  and  foreground 
(threat  response)  goals  was  seen  as  potentially  confusing  and  unable  to  cover  all  combinations  of 
ORO  responsibilities.  However,  proposed  measures  of  performance  were  seen  as  appropriate  and 
feasible  and  are  already  in  use  informally  by  individual  instructors  during  assessments  of  the  ORO 
and  other  positions. 
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The  primary  utility  of  these  reports  was  to  identify  the  critical  situation  awareness  elements  and 
associated  C2  decisions  from  the  perspective  of  the  ORO.  Of  some  surprise,  the  role  of  the  ORO 
was  seen  as  primarily  that  of  a  manager  of  Operations  Room  processes,  rather  than  as  a  primary 
participant  in  tasks  of  picture  building,  threat  and  situation  assessment,  and  response  execution. 
Under  normal  circumstances,  the  ORO  relies  on  each  of  the  warfare  teams  to  adequately  build  and 
assess  the  picture  and  make  recommendations  on  Course  of  Action  (COA).  It  is  only  when  the 
workload  associated  with  one  of  these  activities  exceeds  a  warfare  team’s  capacity  that  the  ORO 
will  act  as  the  primary  information  processor  and  assessor  of  tactical  information,  by  taking  over 
command  responsibility  for  a  warfare  sector. 

In  evaluating  the  utility  of  these  findings  for  the  COMDAT  TD,  it  was  evident  that  the  primary 
information  needs  of  the  ORO  position  were  unlikely  to  be  addressed  by  the  level  1  or  2  fusion 
capabilities  (Roy  2001)  that  were  expected  to  be  the  focus  of  the  TD.  Consequently,  it  was 
recognised  that  many  of  the  information  requirements  associated  with  tactical  picture  compilation 
and  threat  assessment  were  likely  to  be  better  addressed  by  considering  MOPs  that  focussed  on 
warfare  team  members  more  directly  involved  with  those  processes  than  the  ORO. 

One  criticism  of  the  analyses  could  be  that  they  were  based  exclusively  on  concepts  of  situation 
awareness,  decision  making,  communication,  and  workload  derived  from  prior  report 
recommendations  on  how  to  measure  C2  processes.  As  such,  the  issue  of  situation  analysis  appears 
to  have  been  overlooked  as  an  important  area  of  C2  activity.  Further,  the  cognitive  analyses  were 
not  tied  to  specific  theoretical  constructs  such  as  work  domain  analysis  or  hierarchical  goal 
analysis. 

2.2  Relevant  research  from  TTCP  HUM  TP9 

HUM  TP9  is  responsible  for  conducting  collaborative  research,  of  mutual  benefit  to  TTCP  nations, 
aimed  at  integrating  human  factors  into  the  design  of  safe,  manpower  efficient,  and  cost-effective 
systems  for  current  and  future  naval  platforms.  Collaboration  under  TP-9  is  carried  out  in  three 
main  areas: 

•  technology  drivers:  evaluating  the  application  and  implications  of  new  technology  for 
naval  systems, 

•  acquisition  process:  the  insertion  of  HF  into  the  acquisition  process  for  naval  systems,  and 

•  tools  for  optimising  naval  platform  manning  requirements. 

Over  the  past  several  years,  research  conducted  jointly  and  separately  by  member  nations  in  each 
of  these  areas  has  been  made  available  to  all  members.  In  particular,  the  research  on  technology 
drivers  proved  of  immense  value  to  COMDAT.  The  results  supplemented  and  validated  many  of 
the  results  and  recommendations  arising  out  of  the  studies  carried  out  under  COMDAT. 

One  of  the  earliest  collaborative  outputs  from  HUM  TP9  was  a  generic  model  of  the  Human 
Systems  Integration  (HSI)  process.  The  model  emphasises  functional  analysis,  which  is  the  key  to 
HF  application  in  systems  design.  In  conjunction  with  operational  scenarios  and  descriptions  of  the 
operators’  experience,  the  functional  analysis  provides  the  context  of  use  that  is  required  for  the 
development  of  the  operator-machine  interface. 
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Figure  1:  Generic  model  of  the  Human  Systems  integration  process 

The  development  of  the  COMDAT  HF  work  items  did  not  follow  this  model  completely,  but  the 
model’s  influence  can  be  seen  in  the  over  all  pattern  of  research.  The  requirements  analysis  was 
aimed  at  capturing  the  existing  mission,  functions,  and  concepts  of  operation  as  well  as  providing 
insight  into  new  functions  that  would  be  required  (Sections  3  and  7).  Measures  of  performance 
were  developed  to  support  the  evaluation  of  new  decision  support  design  concepts  (Section  4) 
while  the  baseline  analysis  was  necessary  to  do  design  trade-offs  (Section  5).  The  HCI  work  unit 
(Section  6)  was  aimed  at  supporting  the  development  of  HF  prerequisites.  Finally,  the  design  spiral 
was  captured  in  the  proposed  HF  evaluation  of  the  Technology  Demonstrator  (Section  9). 

The  model  was  also  used  as  part  of  a  large  US  project  on  manning  affordability1.  This  project’s 
ultimate  goal  was  to  provide  the  processes,  tools,  interaction  guidelines,  and  procedures  required  to 
optimize  a  combat  systems  environment  for  the  warfighter  at  reduced  manning  levels.  As  with 
COMDAT,  the  project  involved  the  development  of  human  performance  models  and  metrics, 
human-centered  design  tools,  and  OMI  technologies,  but  on  a  much  larger  scale.  Many  of  the  OMI 
technologies  were  implemented  and  evaluated  in  the  Multi  Modal  Watch  Station  (MMWS)  and  the 
Integrated  Combat  Environment  (ICE).  The  former  simulated  a  futuristic  work  station  for  a  combat 


1  Details  on  this  project  which  ran  from  1996  to  2000  can  be  found  at 
http://www.manningaffordability.com/S&tweb/Index  main.htm. 
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systems  operator,  while  the  latter  simulated  a  possible  Operations  Room  incorporating  the  MMWS. 
The  results  from  this  and  related  US  projects  have  proven  of  immense  value  to  COMDAT.  For 
example,  research  on  the  development  of  methods  and  measures  for  assessing  performance 
(Kellmeyer  and  Osga  2000)  supported  recommendations  made  under  COMDAT  for  measuring 
human  performance  in  the  COMDAT  TD.  As  well,  evaluations  of  traditional  and  experimental 
symbol  sets  to  support  more  effective  information  processing  (Kirkpatrick,  Dutra,  Fleasly,  Granda, 
and  Vingelis  1992a;  Kirkpatrick,  Dutra,  Lyons,  Osga,  and  Puggi  1992b;  NAVSPAWARSYSCOM 
1991)  both  informed  and  complemented  the  symbology  studies  carried  out  under  COMDAT 
Finally,  the  Combat  Systems  Functional  Allocation  Board  (CSFAB)  Fluman-System  Interaction 
(HSI)  Style  Guide  (Osga  and  Kellmeyer  2000),  provided  under  HUM  TP9,  was  one  of  the  sources 
for  the  COMDAT  Style  Guide.  Annex  A  provides  a  detailed  summary  of  the  most  relevant  studies. 

In  addition  to  the  specific  knowledge  acquired  through  the  individual  studies,  the  success  of  the 
manning  affordability  project  supported  the  institutionalization  of  a  human  centered  design  process 
into  the  US  Navy  acquisition  process.  Through  a  series  of  HUM  TP9  workshops,  the  Canadian 
Navy  was  made  aware  of  the  approach  and  outcomes  of  this  project  and  as  a  result  has  instituted  a 
more  human-centered  approach  to  acquisition.  One  recent  example  was  the  request  for  a  more 
extended  requirements  analysis  than  originally  planned  for  under  COMDAT.  Finally,  many  of  the 
design  concepts  and  OMI  technologies  evaluated  under  the  manning  affordability  project  are  being 
considered  by  HMCCS. 
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Requirements  Analysis 


HU  MAN. 9  Y  STEMS 


3.1  Goals 

The  goals  of  this  work  were  to  perform  a  requirements  analysis  of  the  operators  in  the  Operations 
Room  of  a  Halifax  Class  frigate.  As  with  any  requirements  analysis,  the  purpose  is  to  document 
the  existing  functions  and  tasks  performed  in  order  to  provide  important  information  for  system 
designers. 

The  positions  of  interest  were  the 

•  Commanding  Officer  (CO), 

•  Operations  Room  Office  (ORO), 

•  Sensor  Weapons  Controller  (SWC), 

•  Assistant  Sensor  Weapons  Controller  (ASWC), 

•  Information  Management  Director  (IMD), 

•  Operations  Room  Supervisor  (ORS), 

•  Warfare  Officer, 

•  Track  Supervisor  (TS), 

•  Electronic  Warfare  Supervisor  (EWS), 

•  Air  Raid  Reporting  Operator  (ARRO),  and 

•  Anti  Submarine  Plotting  Operator  (ASPO). 


3.2  Responsible  parties: 

The  Multi  Function  Task  Analysis  (MFTA)  and  Hierarchical  Goal  Analysis  (HGA)  were 

contracted  to  CMC  Electronics  and  the  Cognitive  Work  Analysis  (CWA)  was  contracted  to 

1  lumansy.vte/w.v  Incorporated. 
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(ORO)  Sensor  Weapons  Controller  (SWC)  and  Assistant  Sensor  Weapons  Controller 
(ASC)  positions:  Mission,  function  and  task  analysis  report.  (DRDC  Toronto  CR  2002- 
219).  CMC  Electronics  Ltd,  Ottawa,  ON. 

Langille,  R.  and  Kelleher,  D.  (2002).  Task  analysis  of  the  Halifax  Class  Sensor  Weapons 
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Ottawa,  ON. 

Coates,  C.  and  Kelleher,  D.  (2002).  Task  analysis  of  the  Halifax  Class  Operations  Room  Officer 
(ORO)  Sensor  Weapons  Controller  (SWC)  and  Assistant  Sensor  Weapons  Controller 

Humansystems'  Incorporated 


Final  COMDAT  Human  Factors  Report 


Page  11 


(ASC)  positions:  Information  flow  and  processing  analysis  report.  (DRDC  Toronto  CR 
2002-220).  CMC  Electronics  Ltd,  Ottawa,  ON. 

Coates,  C.  (2006).  Human  factors  analyses  of  operator  positions  in  the  Operations  Room  of  the 
HALIFAX  Class  frigate.  (DRDC  Toronto  CR  2006-117).  CMC  Electronics  Ltd.,  Ottawa, 
ON. 

Matthews,  M.  L.,  Keeble,  A.R  and  Bruyn  Martin,  L.  (2007).  Analysis  of  the  role  of  Dempster- 
Shafer  algorithms  to  support  recognition  of  surface  and  air  contacts  by  Halifax  Class 
Operations  Room  operators.  (DRDC  Atlantic  CR  2007-091).  Human systems  Inc.,  Guelph, 
ON. 

Matthews,  M.L.,  Keeble,  A.R  and  Sartori,  J.  (2007)  Decision  Ladder  modelling  of  the  recognition 
and  identification  processes  for  surface  and  air  contacts  on  the  Halifax  Class  frigate. 
(DRDC  Atlantic  CR  2007-093).  Yiumansystems  Inc.,  Guelph,  ON. 


3.4  Summary  of  reports 

The  different  requirements  analyses  of  various  positions  in  the  Halifax  Class  Operations  Room 
(Ops  Room),  conducted  prior  to  and  under  COMDAT,  covered  a  very  broad  spectrum  of 
approaches  to  documenting  the  processes  in  building  the  tactical  picture.  The  following  summary 
provides  the  reader  with  a  very  general  understanding  of  what  each  methodology  provided: 

•  Cognitive  Task  Analysis  (CTA):  provided  a  higher  level  view  of  the  cognitive  demands  at 
the  ORO  position.  The  need  for  management  of  the  Ops  Room  was  identified  together 
with  the  problem  of  the  ORO  constantly  having  to  switch  task  and  operational  focus  while 
still  maintaining  both  tactical  and  management  situation  awareness  as  well  the  lack  of 
current  systems  to  support  such  activities  (Matthews  et  al.  1999a;  Matthews  et  al.  1999b). 

•  Mission  Function  Task  Analysis  (MFTA):  provided  a  detailed  description  of  the  tasks  as 
they  are  currently  performed  by  the  SWC,  ASWC  and  ORO.  Critical  tasks  were  identified. 
The  task  set  comprised  both  picture  compilation  and  ship  response  activities.  The  results 
identified  critical  tasks  that  could  be  re-assigned  from  operator  to  machine.  The  level  of 
detail  of  the  approach  did  not  extend  to  a  fine  grain  analysis  of  the  tasks  that  would  inform 
specific  design  solutions  for  specific  operators  (Coates  and  Kelleher  2002b;  Langille  and 
Kelleher  2002). 

•  Information  flow  and  processing  analysis :  provides  operational  sequence  diagrams  and 
identified  the  critical  information  flow  between  the  ORO,  SWC  and  ASWC  and  from  these 
positions  to  other  operators  and  external  agencies  (Coates  and  Kelleher  2002a). 

•  Hierarchical  Goal  Analysis  (HGA):  provided,  for  each  operator  position,  the  goals 
associated  with  what  the  system  needed  to  achieve  in  different  operational  contexts. 
Stability  analysis  revealed  potential  sources  of  instability  and  suggested  design  solutions. 
Information  flow  analysis  (upwards  only)  showed  information  bottlenecks  and  the 
overload  on  the  auditory  communication  channel  (Coates  2006). 

•  Analysis  of  the  role  of  Dempster-Schafer  (D-S)  and  decision  ladder  modelling :  provided  a 
finer-grain  account  of  the  detailed  sequential  activities  involved  in  contact  recognition  and 
identification  by  the  Above  Water  Warfare  (AWW)  team.  Specific  operational  contexts 
where  D-S  may  assist  the  operators  were  identified.  The  specific  information  used  by 
operators  at  each  stage  of  the  process  was  identified  together  with  the  source  and  type  of 
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data  provided  (Matthews,  Keeble,  and  Bruyn  Martin  2007a;  Matthews,  Keeble,  and  Sartori 
2007c). 


3.4.1  Task  analysis  of  the  Halifax  Class  Sensor  Weapons  Controller  (SWC)  and 
Assistant  Sensor  Weapons  Controller  (ASWC)  positions:  Mission,  function 
and  task  analysis  report. 

Task  analysis  of  the  Halifax  Class  Operations  Room  Officer  (ORO)  Sensor 
Weapons  Controller  (SWC)  and  Assistance  Sensor  Weapons  Controller 
(ASWC)  positions:  Mission,  function  and  task  analysis  report. 


Excerpt  from  executive  summaries  (no  abstracts) 

CMC  completed  a  traditional  MFTA  of  the  SWC  and  ASWC  positions,  representing  activities 
through  the  range  of  missions  carried  out  by  those  positions  in  the  HALIFAX  Class  frigate 
during  peacetime,  transition  to  war  and  wartime  operations.  The  aim  of  the  MFTA  was  to 
identify  the  critical  tasks  performed  by  the  SWC  and  ASWC,  and  to  quantify  the  frequency  of 
interaction  and  methods  of  interaction  between  the  SWC/ASWC  and  the  ORO.  CMC  then 
updated  all  work  items  of  the  SWC/ASWC  MFTA  to  identify  the  critical  tasks  performed  by 
the  ORO  and  the  interaction  between  the  ORO  and  higher  Command. 

The  MFTA  of  the  SWC/ASWC  positions  produced  a  task  inventory  of  1 155  tasks.  When  the 
ORO  position  was  added  this  number  grew  to  over  2600.  The  tasks  rated  as  critical  (33%) 
were  subjected  to  a  further  analysis.  These  analyses  of  the  ORO,  SWC  and  ASWC  positions 
establish  a  baseline  database  of  the  process  conducted  by  the  Ops  Room  team  and  therefore 
provide  the  basis  for  assisting  future  system  re-design  efforts. 


3. 4. 1.1  Summary 

Introduction 

The  aim  of  these  reports  was  to  document  the  methodology  and  results  of  the  MFTA.  Tasks  related 
to  decision  making  and  the  transfer  of  information,  particularly  within  the  Ops  Room,  were  the 
primary  focus  of  the  analysis. 

Analysis  methodology 

There  were  4  stages  to  the  analysis: 

•  Mission  analysis, 

•  Function  analysis, 

•  Task  analysis,  and 

•  Critical  task  analysis. 

SMEs  were  used  to  augment  the  domain  knowledge  available  within  CMC. 

Mission  analysis  results 

The  mission  analysis  started  at  a  high  level  looking  at  missions  that  have  been  conducted  by  the 
Canadian  Maritime  Forces  (peacetime  operations,  operations  other  than  war,  and  wartime 
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operations).  Then,  the  mission  analysis  looked  at  the  Halifax  Class  frigate  (weapons,  sensors,  and 
organizational  structure).  Next,  the  focused  shifted  to  the  Ops  Room  and  its  systems,  missions, 
personnel,  and  organization.  Subsequently,  details  on  the  SWC/ASWC  and  ORO’s  responsibilities 
were  outlined.  Finally,  a  composite  scenario  was  created  that  focused  on  the  critical  mission 
segments  performed  by  the  SWC  and  ASWC. 

Function  analysis 

A  function  analysis  identified  the  sequence  of  functions  that  must  be  performed  by  the  system 
through  a  series  of  function  flow  diagrams.  The  detailed  analysis  was  included  in  an  Annex. 

Task  analysis 

A  task  analysis  was  then  performed.  An  Annex  contains  the  detailed  analysis  which  included  the 
following  information  for  each  task  identified: 

•  task  number  and  description; 

•  initiating  conditions; 

•  action,  feedback,  information  and  communication  requirements; 

•  information  available; 

•  decision  component  and  requirements; 

•  frequency; 

•  accuracy  required; 

•  interaction  with  ORO; 

•  Interaction  with  Command  (in  second  report  only); 

•  Task  criticality  rating;  and 

•  Criticality  rationale  (addresses  if  human  performance  limits  are  being  approached;  whether 
or  not  the  analysts  feel  that  further  analysis  may  or  may  not  result  in  improvements). 

Critical  task  analysis 

The  critical  task  analysis  highlighted  the  critical  tasks  relative  to  five  performance  indicators 
(safety,  mission  effectiveness,  efficiency,  system  reliability  and  cost).  28%  of  the  tasks  were 
deemed  ‘critical’  in  the  original  report  that  focused  only  on  the  SWC/ASWC.  When  the  ORO  was 
added  in  the  second  report  the  critical  tasks  increased  to  33%.  In  the  critical  task  analysis  each 
tasks  current  function  allocation  (operator  or  machine)  and  risk  mitigation  was  added  to  the  task 
analysis. 

Critical  task  reallocation 

The  current  function  allocation  was  further  analyzed,  and  reallocation  from  operator  performing  the 
task  to  machine  performing  the  task  was  suggested  for  purposes  of  risk  mitigation.  From  the  first 
report  there  were  seven  tasks  outlined  for  transfer: 

1 .  Conduct  IFF  interrogation. 

2.  Recognize  radar  jamming. 

3.  Identify  chaff. 
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4.  Assess  acoustic  data. 

5.  Initiate  symbology  on  contact. 

6.  Input  harpoon  target  calculations  for  firing. 

7.  Determine  harpoon  panel  calculations  for  launching  harpoon. 

After  the  critical  task  analysis  was  completed  with  the  addition  of  the  ORO,  8  tasks  were  added 
that  if  automated  would  reduce  risk: 

8.  Check  ASW  interfaces. 

9.  Select  optimum  sensor. 

10.  Determine  location. 

1 1 .  Assess  radar  data. 

12.  Assess  effects  of  radar  jamming. 

13.  Recognize  air  contact. 

14.  Establish  position  of  contact  passively. 

15.  Resolve  towed  array  bearing  ambiguities. 

Concluding  material 

All  objectives  of  the  mission,  function  and  task  analysis  of  the  ORO,  SWC  and  ASWC  positions 
were  achieved.  A  number  of  conclusions  and  recommendations  were  made,  including  the 
identification  of  a  pool  of  SMEs  that  are  maintained  throughout  the  study  for  project  continuity 
(  only  in  the  first  report)  and  further  work  to  analyse  the  tasks  and  interactions  of  the  Commanding 
Officer  (only  in  the  second  report). 

3.4.1. 2  Discussion 

These  reports  outlined  a  baseline  list  of  tasks  that  must  be  performed  by  the  ORO,  SWC,  and 
ASWC.  A  scenario  was  also  chronologically  outlined  which  may  prove  useful  for  further  analysis 
or  evaluation  activities.  The  15  specific  tasks,  that  if  re-allocated  from  operator  to  machine  could 
reduce  risk  of  mission  failure,  may  provide  a  logical  starting  place  for  design,  especially  since  they 
have  already  been  validated  by  Navy  SMEs. 

The  analysis  method  chosen  (MFTA)  is  inherently  detailed  oriented.  Thus,  the  design  suggestions 
that  arose  were  at  a  low  level.  A  similar  criticality  analysis  of  higher  level  tasks  may  provide 
design  suggestions  for  other  aspects  of  the  system. 

The  treatment  of  the  SWC  and  ASWC  as  a  single  position  until  the  lowest  level  of  tasks  shows  the 
similarity  between  the  two  positions.  However,  it  does  not  easily  lead  to  any  suggestions 
concerning  the  re-organization  of  responsibilities  between  the  two  operators.  The  level  of 
decomposition  of  the  tasks  does  not  go  down  to  a  level  of  detail  that  would  inform  system 
designers  on  how  a  MSDF  capability  could  be  mapped  onto  the  specific  tasks  of  detection, 
recognition  and  standard  identification  at  the  operator  level.  As  well,  it  would  not  allow  specific 
recommendations  to  be  drawn  on  useful  approaches  to  OMI  design  or  required  changes  in  the 
Concept  of  Operations  (CONOPS). 
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3.4.2  Task  Analysis  of  the  Halifax  Class  Operations  Room  Officer  (ORO)  Sensor 
Weapons  Controller  (SWC)  and  Assistant  Sensor  Weapons  Controller 
(ASWC)  positions:  Information  flow  and  processing  analysis  report. 


Excerpt  from  Executive  Summary  (no  abstract) 

Following  the  results  of  the  MFTA  of  the  ORO,  SWC  and  ASWC,  an  Information  Flow  and 
Processing  Analysis  was  conducted  all  three  positions.  This  report  documents  the  methodology 
and  results  of  the  Information  Flow  and  Cognitive  Processing  Analysis.  It  includes  the 
Operational  Sequence  Diagrams  (OSDs),  which  graphically  depict  mission  segments  against  a 
timeline,  and  an  analysis  of  the  information  and  decision-making  requirements  of  identified 
critical  task  sequences.  Eight  OSDs  were  generated,  depicting  elements  from  all  three  top-level 
functions  or  missions  performed  by  the  HALIFAX  Class  frigate,  Conduct  of  Transit  Operations, 
Conduct  of  Peacetime  Operations  and  the  Conduct  of  Wartime  Operations.  From  these  OSDs,  15 
critical  task  sequences  were  identified,  examining  the  information  flow  between  the  ORO,  SWC 
and  ASWC,  as  well  as  between  these  positions  and  other  operators  in  the  Operations  Room,  other 
departments  within  the  ship,  and  external  agencies. 


Summary 

Introduction 

Three  objectives  were  outlined:  (1)  develop  network  models;  (2)  identify  critical  task  sequences; 
and  (3)  conduct  information  Flow  and  Processing  Analysis  for  the  identified  task  sequences. 
There  are  6  sections  to  the  report,  not  including  Annexes. 

Analysis  methodology 

A  network  simulation  tool  within  IPME  was  used  to  develop  a  model  of  the  functions  performed 
by  humans  in  conducting  mission  segments  from  the  composite  scenario  (created  during  the 
MFTA).  The  network  model  consisted  of  3  Operational  Sequence  Diagrams  (OSDs).  A 
performance  prediction  analysis  was  also  carried  out.  Design  symbology  for  the  OSD  was  also 
outlined  in  this  section  along  with  the  methodology  used  for  the  information  flow  and  processing 
analysis. 

Network  model 

The  roles  and  responsibilities  of  the  participants  of  the  network  model  (ORO,  SWC,  ASWC)  were 
described.  The  use  of  external  agencies  for  the  network  modelling  was  explained,  followed  by 
explanations  on  the  development  of  the  OSDs.  The  network  model  was  only  used  on  mission 
segments  that  were  determined  likely  to  lead  to  the  identification  of  decision-making  and 
information  flow  issues  and  requirements  for  the  Ops  Room  team.  Annex  B  of  the  report  contains 
the  8  OSDs  covering  3  different  mission  types: 

•  OSDs  involved  with  ‘Conduct  transit  operation’, 

•  an  OSD  outlining  ‘Conduct  peacetime  operations’,  and 

•  OSDs  associated  with  ‘Conduct  warfare  operations’. 
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Information  flow  and  cognitive  processing  analysis  results 

The  information  flow  requirements  and  decision  making  process  requirements  to  accomplish 
mission  objectives  were  analysed  in  15  critical  task  sequences.  These  15  task  sequences  were 
selected  based  on  a  subjective  assessment  of  their  importance  to  the  mission  segment  and  the 
likelihood  of  identifying  requirements/issues.  Each  task  has  an  associated  ‘Summary  of  findings’ 
that  lists  recommendations  and  general  conclusions  specific  to  the  task  sequence. 

Concluding  material 

Conclusions  were  made  based  on  the  information  flow  analysis  and  observations  made  by  the 
CMC  team  during  this  phase  of  the  project.  Additionally,  there  were  7  recommendations  that  are 
summarized  below: 

•  Complete  an  information  flow  analysis  of  all  Ops  Room  personnel  (at  minimum  CO, 
Officer  of  the  Watch  (OOW),  ORS). 

•  Carry  out  a  quantitative  performance  analysis  of  key  Ops  Room  personnel. 

•  Include  the  HALIFAX  Class  frigate  employed  as  a  Command  ship  in  future  analysis. 

•  Include  SMEs  serving  on  the  West  Coast  in  further  analysis. 

•  Investigate  the  feasibility  of  developing  a  detailed  checklist/handbook  for  each  position  in 
the  Ops  Room  (Canadian  Forces  Naval  Operations  School  (CFNOS)  suggested  to  create). 

•  Conduct  Maritime  Interdiction  Operations/Information  Operations  modelling. 

•  Examine  the  physical  layout  of  the  Ops  Room  and  operator  positioning  for  optimization. 

Discussion 

This  report  adds  to  the  baseline  information  already  collected  concerning  Ops  Room  personnel 
tasks.  Though  the  final  conclusions/recommendations  included  in  the  main  body  of  the  report  do 
not  outline  suggestions  on  changes  to  the  current  Ops  Room,  the  actual  analysis  contained  in 
Annex  C  (Information  Flow  and  Operational  Sequence  Analysis)  does  provide  a  number  of  specific 
suggestions  to  optimize  performance  of  the  Ops  Room.  Most  of  the  changes  are  small,  but 
collectively  they  could  provide  significant  improvements. 
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3.4.3  Human  factors  analyses  of  operator  positions  in  the  Operations  Room  of  the 
HALIFAX  Class  frigate. 


Abstract 

Results  are  provided  for  the  analyses  of  eleven  operator  positions  in  the  Halifax  Class  frigate 
Operations  Room  using  the  Hierarchical  Goal  Analysis  (HGA)  approach.  Following  mission 
analyses,  a  hierarchy  of  goals  assigned  to  different  operators  was  produced.  Two  follow-on 
analyses  were  conducted  to  identify  potential  instabilities  in  the  system  and  requirements  for 
upward  information  flow  between  operators.  Operational  Sequence  Diagrams  (OSDs)  were 
produced  for  five  critical  task  sequences  and  the  corresponding  task  networks  were 
implemented  and  tested  in  the  Integrated  Performance  Modeling  Environment  (IPME).  The 
final  product  of  the  project  was  the  generation  of  a  list  of  critical  Operations  Room  activities 
supported  by  proposed  solutions.  The  report  concludes  HGA  and  IPME  are  suitable  tools  to 
support  the  analyses  of  complex  predominantly  ‘cognitive’  systems. 


Summary 

Analysis  methodology 

Four  different  analyses  were  conducted  (with  assistance  from  SMEs)  on  1 1  operator  positions  in 
the  Halifax  Class  frigate: 

1.  Hierarchical  Goal  Analysis  (HGA):  models  a  cognitive  system  consisting  of  one  or  more 
operators  by  identifying  goals,  at  various  levels  of  abstraction,  which  the  system  needs  to 
achieve.  Goals  are  desired  states  to  which  current  states  are  compared. 

2.  Information  Processing/Perceptual  Control  Theoiy  (IP/PCT)  Model:  represents  the 
operator’s  allocation  of  attention  and  human  memory  together  with  a  framework  for 
tracking  the  load  on  the  operator’s  information  processing  system.  The  model  was 
implemented  in  IPME.  The  IP/PCT  data  was  then  used  to  generate  the  upward  information 
flow  analysis  (there  is  a  need  for  information  flow  when  one  operator  controls  a  goal  that 
supports  a  goal  controlled  by  another  operator)  and  stability  analysis  (looking  to  see  if 
more  than  one  operator  controls  the  same  external  variable). 

3.  Task  networks:  The  IPME  model  requires  a  chronological  set  of  events  in  order  to  run  the 
program  and  identify  periods  of  operator  loading.  The  networks  represented  five  critical 
task  sequences. 

4.  Critical  activities:  A  modified  Cooper-Harper  Scale  was  used  to  rate  goals  using  criteria 
for  safety,  mission  effectiveness,  and  human  performance  capability.  The  goals  with 
higher  values  in  any  of  the  rating  criteria  were  then  further  analyzed  to  provide  potential 
solutions. 

After  a  detailed  description  of  each  type  of  analysis,  the  report  included  a  brief  description  of  the 
analysts’  interaction  with  SMEs  as  well  as  an  environmental  analysis  that  summarized  the 
environment  in  which  the  Halifax  Class  frigate  is  required  to  operate. 
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Hierarchical  Goal  Analysis 

There  were  a  total  of  562  goals  outlined.  Complete  listings  of  the  top  level,  first,  and  second  level 
goals  are  included  in  the  main  body  of  the  report.  Goal  allocation  and  usage  are  included  in  the 
Annex.  The  report  notes  that  a  ‘bottom-up’  review  of  the  goal  decomposition  should  be  conducted 
to  validate  the  top-down  process  used  by  the  analysts  in  the  HGA. 

Junior  operators  tend  to  repeat  goals  more  frequently  in  their  day-to-day  activities,  so  while  they 
are  as  occupied  or  as  busy  as  the  senior  operators,  they  are  allocated  fewer  unique  goals.  The  SWC 
has  the  most  goals  because  as  a  director  he/she  is  responsible  for  surface  and  air  picture 
compilation  and  warfare.  In  a  general-purpose  frigate  this  effectively  doubles  the  SWC’s 
responsibilities  in  comparison  to  the  ASWC,  who  is  only  responsible  for  subsurface  picture 
compilation  and  warfare. 

IP/PCT 

For  each  of  the  562  goals,  a  Perceptual  Control  Theory  (PCT)  table  was  created.  Completing  the 
PCT  analysis  was  challenging;  especially  the  resolution  of  the  internal,  external  and  controlled 
variables.  The  technical  authority  was  used  in  place  of  SMEs  for  review  of  the  tables  due  to  their 
volume  and  the  corresponding  time  required  to  fully  review  them.  In  the  end,  the  PCT  data  proved 
extremely  useful  for  subsequent  analyses. 

Stability  analysis 

Given  the  large  and  complex  nature  of  the  Halifax  Class  frigate,  the  number  of  external  variables 
identified  as  being  controlled/influenced  by  more  than  one  operator  (13  variables)  is  low.  A 
detailed  discussion  is  included  that  outlines  additional  potential  sources  of  instability  (e.g.  shared 
responsibilities,  conflicting  goals,  etc.)  and  suggests  solutions.  For  example 

•  Limit  access  to  equipment  controls  to  the  primary  and  secondary  owner  of  the  goal 
responsible  for  those  specific  settings. 

•  Implement  shared  situation  awareness  displays. 

•  Ensure  visibility  of  the  (ideally  electronic)  stateboard  to  all  operators  who  require 
information. 

•  Lock  out  non-critical  users  from  CCS. 

•  Include  a  greater  capacity  for  overlays,  including  the  ability  to  preview  the  screen  to 
visualize  a  planned  overlay. 

•  Increase  use  of  real  time  text  based  communications. 

The  report  also  noted  that  a  complete  list  of  instabilities  was  not  included  due  to  demand  on  the 
operators  and  future  complex  organizational  analyses  should  include  all  operators  that  support  the 
completion  of  a  goal  in  the  instability  analysis. 

Upward  information  flow  analysis 

Different  presentation  methods  for  the  upward  information  flow  were  tested  throughout  the  project 
and  resulted  in  the  development  of  an  interactive  viewer  that  optimized  data  display.  This  display 
was  presented  to  the  SMEs,  and  observations  recorded.  A  major  observation  was  that  the  volume 
of  verbal  reports  puts  an  overwhelming  demand  on  the  auditory  channel,  while  the  visual  stimuli 
are  quite  limited.  A  limitation  noted  in  the  report  is  that  only  an  upward  information  flow  analysis 
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was  conducted,  but  lateral  and  downward  communications  should  be  added  to  provide  a  complete 
link  analysis. 

Task  networks 

OSDs  were  developed  on  a  goal  by  goal  basis  and  provided  representations  of  the  critical 
operational  sequences  to  SMEs.  After  validation,  the  tasks  were  run  through  IPME  where  many 
technical  glitches  were  discovered  -  mostly  related  to  the  IPME  scheduler  -  and  a  full  system 
implementation  was  not  possible.  Single  runs  and  multi-threat  scenarios  were  successfully 
conducted,  and  the  results  presented.  For  SME  validation,  the  IPME  outputs  were  not  as  useful  as 
desired.  A  lack  of  OSDs  of  the  entire  network,  no  straightforward  ways  to  combine  outputs  from 
different  runs,  and  novel  terminology/concepts  all  made  SME  validation  of  IPME  outputs  difficult. 

Critical  activities 

No  single  operator  or  group  of  operators  had  a  monopoly  on  critical  activities.  The  CO  had  the 
highest  percentage  of  activities  considered  critical,  followed  by  the  ASPO  and  EWS  respectively. 

In  discussions  with  SMEs,  63%  of  activities  were  considered  for  proposed  solutions.  The  majority 
of  solutions  recommended  automating  displays  and  data  entry,  as  well  as  the  integration  of  tactical 
command  decision  aids. 

Conclusions/recommendations 

The  analysis  methodologies  were  found  to  be  applicable  to  the  complexities  of  the  Halifax  Class 
Ops  Room.  As  such,  the  information  collected  during  this  project  was  useful  for  the  OMI 
specifications  for  the  upgrade  to  the  Ops  Room  and  was  applied  to  the  redesign  of  the  operator 
workstations,  operator  interface,  and  Ops  Room  layout. 

Discussion 

This  report  provides  a  large  number  of  practical  and  specific  suggestions  concerning  the 
advancement  of  the  current  Ops  Room  on  a  HALIFAX  Class  frigate.  The  stability  analysis,  upward 
information  flow  analysis,  task  networks,  and  critical  activities  analysis  all  resulted  in  functional 
change  suggestions.  The  extent  to  which  the  specific  suggestions  are  effective  cannot  be  shown 
yet;  however,  the  issues  identified  are  based  on  a  thorough  analysis  validated  by  SMEs.  It  should 
be  noted  that  none  of  the  listed  solutions  mention  how,  or  if,  multi-sensor  data  fusion  would  be 
advantageous. 

The  SMEs’  difficulty  in  using  IPME  output  to  validate  the  model  was  noted  which  points  to  the 
need  to  make  such  output  more  usable  to  the  operational  community. 

The  OSDs  included  in  the  annexes  provide  a  detailed  understanding  of  the  actions  the  different 
operators  are  required  to  perform,  including  the  recognition/identification  process. 

Communications  between  different  operators  are  clearly  shown.  Analysts  wanting  a  better 
understanding  of  the  Ops  Room  are  encouraged  to  review  the  material.  The  level  of  decomposition 
and  analysis  for  the  conduct  of  recognition  and  identification  process  is  probably  insufficient  to 
inform  design  aids  for  these  processes  at  the  individual  operator  level.  Readers  should  consult 
Matthews,  Keeble  and  Bruyn  (2007)  and  Matthews,  Keeble  and  Sartori  (2007)  for  this  kind  of 
detail. 
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3.4.4  Analysis  of  the  role  of  Dempster-Shafer  algorithms  to  support  recognition  of 
surface  and  air  contacts  by  Halifax  Class  Operations  Room  operators 


Abstract 

This  report  integrates  thirteen  work  items  completed  under  the  COMmand  Decision  Aid 
Technology  Demonstration  Project  (COMDAT).  The  goal  of  these  work  items  was  to  document 
the  existing  recognition  and  identification  processes  for  air  and  surface  contacts  performed  by 
the  Above-Water  Warfare  (  AWW)  team  of  the  HALIFAX  Class  frigate.  The  report  provides  an 
analysis  of  the  potential  role  that  Dempster-Shafer  (D-S)  algorithms  for  evidential  reasoning 
might  play  to  support  contact  recognition  and  identification.  Finally,  potential  air  and  surface 
scenario  events  where  D-S  may  assist  the  team  were  devised. 


Summary 

This  work  addresses  the  requirement  to  investigate  and  develop  a  concept  of  use  that  would 
potentially  allow  D-S  functionality  for  evidential  reasoning  to  be  deployed  in  the  CCS  to  provide 
effective  support  to  Ops  Room  operators  in  the  conduct  of  recognition  and  standard  identification 
of  air  and  surface  contacts.  The  investigation  included  operator  roles,  information  flows,  operator- 
automated  system  interaction  schemes,  and  interface  design. 

The  report  provides  a  detailed  analysis  of  the  recognition  and  identification  processes  as  they  are 
currently  conducted  for  air  and  surface  contacts  based  on  the  ship’s  current  sensor  suite  and  off- 
board  data  sources  as  well  as  its  current  information  acquisition  and  integration  procedures. 

The  phases  of  the  work  included  the  development  of  a  technical  plan,  interviews  with  Navy  SMEs, 
analysis  of  the  results,  graphing  and  tabulation  of  information  flow  sequences,  and  evaluating  the 
role  that  might  be  played  by  D-S  algorithms  to  support  evidential  reasoning  by  the  ship’s  team. 

Within  the  constraints  on  the  existing  analyses,  the  data  did  not  reveal  operational  situations 
involving  detection,  identification,  and  recognition  where  D-S  functionality  (as  it  was  considered  to 
be  implemented  in  the  TD)  could  be  seen  to  have  a  role  in  enhancing  the  normative  processes 
involved.  There  are  some  circumstances,  where  crew  attention  is  elsewhere,  or  there  are  high  rates 
of  contacts,  that  may  point  to  a  limited  role  whereby  D-S  recommendations  could  enhance  situation 
awareness  and  decision  making. 

The  analyses  suggest  that  the  major  processes  of  evidential  reasoning,  which  prove  challenging  for 
the  crew,  are  in  the  association  of  EW  information  with  radar  data,  and  the  assessment  of 
contextual  information.  Therefore,  it  was  suggested  that  it  was  these  aspects  of  analysis  and 
decision  making  that  could  be  fruitfully  explored  in  the  context  of  a  more  generic  MSDF  capability 
that  embraces  broader  information  usage  than  currently  implemented  in  the  TD. 

The  report  also  identified  a  potentially  important  role  in  recognition  for  D-S  and  MSDF  with  more 
generic  capabilities  than  were  implemented  in  the  TD.  Such  a  role  has  been  outlined  in  some  detail 
in  Keeble,  Matthews,  Lamoureux,  Berger,  and  Zobarich  (2007). 

Finally,  the  circumstances  of  twelve  surface  and  twelve  air  events  were  documented  that  could  give 
rise  to  the  useful  application  of  MSDF  and/or  D-S  technology.  The  events  were  created  through  a 
“brainstorming”  process  but  are  based  to  some  extent  on  SME  experience  and  interview  data.  Two 
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of  the  events  were  then  selected  to  show  in  detail  how  MSDF  and  D-S  functionality  could  be 
implemented  to  support  team  recognition. 

Discussion 

These  studies  provide  a  very  detailed  account  of  the  recognition  and  identification  of  air  and 
surface  contacts  by  the  Ops  Room  Command  Team  using  a  timeline  analysis.  The  level  of  detail  in 
describing  these  processes  and  the  granularity  of  analysis  are  finer  than  with  other  analyses 
produced  under  the  COMDAT  TDP.  The  analysis  showed  somewhat  surprisingly  that  there  was 
not  much  evidence  to  show  that  D-S  algorithms  that  incorporated  level  0  and  level  1  fusion 
concepts  (Roy  2001)  would  support  the  typical  evidential  reasoning  undertaken  by  operators  on 
single  contacts.  When  evidential  reasoning  is  used,  it  tends  to  involve  the  use  of  information  based 
upon  context  and  behavioural  characteristics  of  contacts,  or  involves  decisions  concerning  the 
association  of  EW  information  with  platform  tracks.  Thus  in  order  for  D-S  algorithms  to  have  a 
useful  role  in  contact  recognition,  some  form  of  level  two  evidential  reasoning  would  need  to  be 
incorporated,  as  well  as  an  ability  for  algorithms  to  have  access  to  information  that  is  available  to 
the  team  but  does  not  come  through  MSDF  sensor  sources. 

However  in  spite  of  this  finding,  the  study  goes  on  to  show  some  recognition  situations  where  D-S 
based  evidential  reasoning  algorithms  could  assist  a  team  that  was  inattentive  or  otherwise 
overloaded.  The  twelve  specific  air  and  surface  situations  identified  could  represent  some 
interesting  test  cases  to  evaluate  the  performance  of  any  future  evolution  of  the  HALIFAX  Class 
CCS  for  its  ability  to  support  the  recognition  process. 

3.4.5  Decision  Ladder  modelling  of  the  recognition  and  identification  processes 
for  surface  and  air  contacts  on  the  HALIFAX  Class  frigate. 


Abstract 

This  report  presents  results  of  an  analysis  of  the  surface  and  air  picture  compilation  processes 
by  the  Operations  Room  personnel  of  a  HALIFAX  Class  frigate  using  a  decision  ladder 
modelling  approach.  The  analysis  focused  on  the  Recognition  and  Standard  Identification  of  a 
single  air  and  surface  contact  and  the  management  of  these  processes.  The  goal  of  the  analysis 
was  to  identify  the  information-processing  activities  operators  need  to  perform  in  recognition 
and  standard  identification,  the  major  short-cuts  between  these  processing  activities,  and  the 
knowledge  states  that  result  from  these  activities.  This  was  done  to  inform  future  design 
considerations  to  support  the  integration  of  multi-sensor  data  fusion  technologies  into  future 
enhancements  to  the  HALIFAX  Class  Command  and  Control  System. 


Summary 

This  project  focussed  on  the  picture  compilation  process  for  air  and  surface  targets  using  a 
Decision  Ladder  (DL)  analysis  methodology  to  uncover  the  primary  processing  activities  and 
knowledge  states.  The  DL  analysis  examined  specifically  the  recognition  of  a  single  air  or  surface 
contact  and  the  management  of  these  processes. 

The  DL  approach  requires  an  organisation  of  the  material  into  a  ladder  structure  as  indicated  in  the 
standard  template  for  a  DL  representation  in  the  figure  below.  An  important  part  of  the  DL  analysis 
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is  the  identification  of  “leaps”  and  “shunts”,  which  represent  short-cuts  that  are  taken  by  skilled 
and  experienced  operators. 


Figure  2:  Representational  template  for  a  generic  Decision  Ladder 

The  analysis  also  provided 

•  a  description  of  the  “system”, 

•  the  goals,  and 

•  performance  metrics  for  evaluating  system  states,  goals,  and  performance  and  resolving 
competing  goals. 

In  addition,  the  detailed  descriptions  of  each  processing  activity  included  information  on  its 

•  purpose, 

•  information  used,  and 

•  a  priori  information  used 
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Direction  from  the  Scientific  Authority  indicated  that  the  approach  to  be  taken  was  a  formative 
description,  rather  than  a  serial  analysis  of  the  sequential  process  steps.  In  particular,  one  goal  of 
the  analysis  was  to  identify  short-cuts  in  the  process  that  might  identify  opportunities  for  future 
design  improvements  to  the  CCS  to  integrate  MSDF  and  D-S  capabilities. 

The  information  used  to  conduct  the  analyses  was  based  on  interviews  with  Navy  SMEs,  other 
analyses  conducted  in  support  of  COMDAT,  and  relevant  Navy  documentation. 

The  first  step  in  the  analysis  involved  creating  a  detailed  inventory  of  the  information  sources;  this 
involved  providing  details  on  the  following: 

•  source  information 

•  recognition,  identification  and  derived  information 

•  significance  of  context  and  lack  of  information,  and 

•  presentation  format 

Following  this  initial  step,  the  decision  ladders  for  recognition  and  identification  of  an  air  contact 
were  evolved.  These  formed  the  basis  for  developing  a  similar  representation  for  a  surface  contact. 
Finally,  the  DL  for  the  management  process  was  created,  a  task  made  simpler  by  the  prior  creation 
of  the  other  DLs. 

The  analyses  revealed  a  number  of  short-cuts  in  every  DL,  typically  between  the  system  state  and 
tasks  or  procedures.  This  outcome  reflected  the  fact  that  the  process  involved  highly  trained  and 
experienced  personnel,  who  could  quickly  go  from  an  awareness  of  the  state  of  the  system  to  the 
required  action  demanded  by  that  awareness.  However,  some  aspects  of  the  management  process 
did  require  an  evaluation  of  options  and  a  weighing  of  action  alternatives,  thereby  involving  more 
nodes  in  the  DL.  For  all  of  the  analyses,  the  step  of  formulating  a  procedure  was  typically  not  seen 
to  be  an  integral  component  of  the  behaviour  of  experienced  personnel. 

Discussion 

This  report  provides  a  highly  detailed  account  of  the  processes  and  knowledge  states  that  support 
recognition  and  standard  identification  of  an  air  and  surface  contact.  The  strengths  of  the  analyses 
are 

1 .  They  provide  in  one  place  a  very  coherent  description  of  what  is  done  in  the  recognition 
and  standard  identification  processes,  and  what  specific  information  is  used  at  each  step  of 
the  processes,  together  with  the  source(s)  of  that  information.  Therefore,  the  analysis 
provides  some  value-added  over  traditional  methods. 

2.  The  information  source  inventory  provides  a  valuable  reference  for  those  wishing  to 
understand  what  information  is  provided  by  a  specific  source  that  contributes  to 
recognition  and  identification  including  the  limitations  and  value  of  that  information. 

3.  The  analyses  show  in  a  generic  sense  what  processes  are  required  and  what  can  be  skipped 
over  (shortcuts)  in  the  conduct  of  recognition  and  standard  identification. 

4.  The  formative  nature  of  the  analyses  allows  for  an  understanding  of  what  generic  support 
could  be  provided  in  the  future  to  enhance  the  Halifax  Class  CCS. 

The  major  limitations  of  the  analysis  are 
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1.  Because  the  analyses  provide  a  generic  description  of  what  is  done,  they  cannot  inform  the 
support  requirements  for  a  specific  operator  doing  a  particular  task  at  a  fixed  point  in  time 
in  recognition  or  identification. 

2.  It  is  also  not  clear  how  the  DL  approach  allows  for  a  simple  representation  of  the  links 
between  functions.  Within  each  function  there  are  multiple  tasks  and  processes,  each  of 
which  may  possibly  link  to  a  task  or  process  in  another  function. 

3.  In  view  of  the  multiple  prior  analyses,  it  is  not  clear  what  new  information  was  provided  by 
the  DL  approach.  While  the  DL  method  has  resulted  in  a  tight  integration  and 
documentation  of  the  processes  performed,  the  knowledge  states  and  information  used  and 
its  sources,  it  is  not  at  all  obvious  what  insights  have  been  gained  as  a  result  of  this  re¬ 
mapping.  For  example,  even  the  identification  of  shortcuts  was  pre-empted  by  some  of  the 
results  of  the  more  serial  analyses  using  information-flow  or  process  sequence  methods 
discussed  previously. 
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4  Development  and  evaluation  of  measures 

of  performance 

4.1  Goals 

The  goals  of  this  work  were 

•  to  use  the  outcomes  of  existing  cognitive  and  other  task  analyses  to  identify  useful  MOPS 
to  evaluate  critical  C2  processes,  centred  on  the  ORO; 

•  to  evaluate  methodologies  and  environments  that  would  be  suitable  for  creating  reliable 
and  valid  MOP  data;  and 

•  to  conduct  feasibility  and  proof-of-concept  assessments  of  the  suitability  of  methodologies 
and  environments  identified  as  potential  approaches  to  MOP  data  collection. 

4.2  Responsible  parties 

All  of  this  work  was  contracted  to  Humansvstojw  "'  Incorporated. 

4.3  Reports  Produced 
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4.4 


Summary  of  reports 


4.4.1  Evaluation  of  potential  test  environments  for  assessing  the  impact  of  multi¬ 
sensor  data  fusion  on  command  and  control  operations  in  the  Halifax  Class 
frigate. 


Abstract 

This  report  reviewed  five  training  simulator  facilities  in  the  Canadian  Navy  for  their  suitability 
in  conducting  test  and  evaluation  (T&E)  in  support  of  the  COMDAT  multi-sensor  data  fusion 
(MSDF)  initiative.  The  facilities  were  the  Operations  Room  Team  Trainer  (ORTT),  the  Naval 
Combat  Operator  Trainer  (NCOT),  the  Combat  Systems  Training  Centre  (CSTC)  and  Halifax 
Class  Operations  Room  at  harbourside  or  at  sea. 

It  was  concluded  that  the  NCOT  would  be  the  most  suitable  facility  for  conducting  T&E  using 
single  operators  performing  simple  operational  functions  such  as  detecting  and  tracking  radar 
contacts. 

For  more  complex  functions  in  the  Ops  Room,  including  tasks  involving  interactions  between 
complete  warfare  teams,  interactions  between  warfare  directors  and  the  ORO,  and  between  the 
ORO,  CO  and  the  entire  Ops  Room  team,  the  report  found  that  the  ORTT  represents  a  more 
suitable  environment  for  conducting  T&E  activities. 


Summary 

A  systematic  evaluation  was  conducted  of  potential  trial  environments  suitable  for  the  execution  of 
COMDAT  related  T&E  trials  with  a  goal  of  collecting  real-time,  human- in-the  loop  (HIL) 
performance  data.  A  set  of  detailed  capabilities  requirements  for  conducting  T&E  was  developed 
under  the  four  major  categories  of:  logistics,  experimental  control,  data  collection  and  storage,  and 
analysis.  Each  of  the  potential  facilities  was  reviewed  and  evaluated  against  these  requirements  and 
the  results  tabulated.  The  review  took  into  account  earlier  work  developing  a  framework  for 
evaluating  military  information  systems  (Matthews,  Webb,  and  McCann  1997)  as  well  as  data  fusion 
levels  (Roy  2001).  This  framework  considered  the  need  to  conduct  iterative  evaluations  of  human 
performance  during  development  in  terms  of  part,  whole,  and  team  tasks  at  several  levels  of 
prototype  design  under  mock  up,  test  bed  or  simulator,  and  sea  trial  conditions.  Existing  Navy 
systems  used  for  collecting  performance  data  were  also  reviewed.  The  capabilities  of  each  facility 
were  then  assessed  for  their  suitability  for  conducting  the  expected  range  of  T&E  activities. 

It  was  concluded  that  the  NCOT  would  be  the  most  suitable  facility  for  conducting  T&E  using 
single  operators  performing  simple  operational  functions  such  as  detecting  and  tracking  radar 
contacts.  While  more  limited  in  its  simulation  fidelity  than  the  ORTT,  the  NCOT  facility  appeared 
to  be  the  most  versatile  to  configure  and  the  most  economical  to  operate.  NCOT  allowed  the 
collection  of  T&E  data  for  several  operators  performing  the  same  task  in  parallel.  NCOT  also 
appeared  suitable  for  assessing  performance  of  small  teams  of  2-4  individuals  performing  more 
complex,  interacting  functions.  This  assessment  was  based  upon  the  expectation  that  the  contractor 
responsible  for  NCOT  would  deliver  software  that  supports  this  level  of  functionality  within  the 
next  year. 
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For  more  complex  functions  in  the  Ops  Room,  including  tasks  involving  interactions  between 
complete  warfare  teams,  interactions  between  warfare  directors  and  the  ORO,  and  between  the 
ORO,  CO  and  the  entire  Ops  Room  team,  the  ORTT  represented  the  preferred  environment  for 
conducting  T&E  activities.  A  range  of  scenarios  has  been  developed  for  team  training  in  the 
ORTT,  which  could  be  adaptable  for  T&E  purposes. 

Severe  constraints  on  the  availability  of  the  ORTT  (and  possibly  the  NCOT)  were  noted,  due  to 
ongoing  training  demands  from  the  Navy.  Further,  the  necessary  personnel  infrastructure  to 
support  trials,  (e.g.  game  players,  controllers  and  communicators)  in  the  ORTT  is  high  and  is 
constrained  by  personnel  availability.  Such  limitations  may  preclude  the  possibility  of  conducting 
dedicated  T&E  trials  in  the  ORTT. 

Discussion 

The  assumption  behind  these  investigations  of  potential  test  environments  was  that  they  could  be 
made  available  for  T&E  purposes  to  evaluate  the  impact  of  COMDAT  technology  on  real-time 
human-system  performance  in  the  conduct  of  critical  C2  tasks.  Of  the  various  options  that  seemed 
feasible  for  such  purposes,  it  was  recommended  that  proof  of  concept  trials  be  conducted  in  both 
NCOT  and  the  ORTT  to  assess  their  actual  capability  and  limitations  using  a  scenario-based 
approach.  In  particular,  there  would  be  a  need  to  demonstrate  a  capability  to  reliably,  accurately 
and  effectively  capture  MOP  data  of  interest. 

Limitations 

There  is  very  little  discussion  of  the  possibility  of  inserting  the  COMDAT  Technology 
Demonstrator  into  these  environments.  In  part  this  was  due  to  a  lack  of  a  good  understanding  of 
what  the  COMDAT  TD  would  look  like  and  be  capable  of  doing  at  the  time  the  report  was 
prepared.  In  addition,  the  report  would  have  benefited  from  a  general  discussion  of  what  form  of 
human  centred  T&E  might  be  conducted  in  the  different  facilities  given  that  the  required 
capabilities  did  not  exist. 

4.4.2  Development  of  measures  of  performance  for  evaluating  the  COMDAT 
technology  demonstrator:  Potential  use  of  the  Naval  Combat  Operator 
Trainer  (NCOT)  for  data  collection. 


Abstract 

This  report  compiles  the  information  contained  in  five  other  reports  and  Technical  Memoranda 
pertaining  to  the  development  of  Measures  of  Performance  (MOPs)  for  Multi-Sensor  Data 
Fusion  technology.  The  report  evaluates  the  logistics  and  the  utility  of  the  Naval  Combat 
Operator  Trainer  (NCOT)  to  gather  these  MOPs.  The  report  recounts  the  development  of 
specific  MOPs,  the  conduct  of  a  proof-of-concept  trial  at  NCOT  (with  attendant  identification  of 
potential  system  improvements),  the  outcome  of  discussions  with  the  contractor  responsible  for 
the  maintenance  and  operation  of  NCOT  regarding  enhancements,  and  the  investigation  of  one 
potential  commercial-of-the-shelf  solution  for  some  of  the  improvements  described.  As  a  result 
of  identifying  these  potential  improvements,  it  was  recommended  that  team  records  created 
during  Navy  team  training  in  Operations  Room  Team  Trainer  (ORTT)  be  considered  as  a  better 
source  for  deriving  MOPs. 
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Summary 

A  series  of  studies  was  conducted  to  evaluate  the  NCOT  environment  as  a  suitable  candidate  for 
gathering  MOPs  to  evaluate  the  COMDAT  technology  against  baseline  performance.  The  first 
study  (also  published  as  Matthews,  Webb,  and  Keeble  (2002))  developed  a  set  of  MOPs  based  on 
the  cognitive  task  analysis  outlined  in  Section  2  (Matthews  et  al.  1 999b)  and  a  comprehensive  plan 
for  evaluating  the  impact  of  future  command  decision  aid  technologies  on  human  and  operational 
performance  in  the  Halifax  Class  Ops  Room.  The  plan  outlined  an  incremental  approach  to  T&E 
to  generate  reliable  and  valid  performance  data  for  a  range  of  critical  tasks  performed  by  the  Ops 
Room  team,  with  a  general  focus  on  the  role  of  ORO. 

A  series  of  data  collection  trials  were  proposed  that  would  match  the  developmental  sequence  of 
MSDF  and  other  technologies  that  formed  the  basis  of  the  COMDAT  TD.  This  sequence  was  to 
involve  (i)  an  initial  concentration  on  the  integration  of  above  water  warfare  (AWW)  sensors  and 
Link  11,  followed  successively  by  (ii)  integration  of  surface  aspects  of  AWW  sensors  and  Link  11, 
(iii)  integration  of  all  UWW  sensors,  and  (iv)  all  of  the  former  integrated  with  the  wide  area  picture 
(WAP). 

For  each  trial  a  detailed  description  was  provided  of  the  logistics  (personnel,  software  support  and 
facilities)  that  would  be  required  to  prepare,  conduct  and  analyse  the  trial.  Trials  would  be 
scenario-based,  with  the  scenario  content  largely  focussed  on  the  warfare  area  that  the  TDP  was 
designed  to  support.  The  trial  sequence  would  involve  the  initial  collection  of  baseline  performance 
data  using  existing  technologies  to  serve  as  a  subsequent  comparison  for  the  same  operational  tasks 
aided  by  the  COMDAT  technology.  Requirements  for  the  test  scenario  were  provided  in  detail 
including  the  geographical,  environmental,  political  and  military  contexts,  the  game  entities  and 
their  dynamics,  and  the  sequence  and  number  of  events.  It  was  recommended  that  these  initial 
trials  be  conducted  in  the  NCOT  facility,  which  appeared  to  provide  many  of  the  requirements  for 
the  first  stage  of  evaluating  the  most  immediately  emerging  TDP  functionality,  while  minimising 
the  logistical  overhead  to  support  the  trial. 

The  initial  effort  focused  on  MOPs  for  AWW  as  well  as  the  generic  situation  awareness  needs  of 
the  ORO  in  assessing  threats  and  building  the  tactical  picture.  It  also  provided  details  of  the 
research  design  in  terms  of  the  number  of  data  trials  and  test  participants  that  would  be  needed  to 
achieve  the  required  sensitivity  to  detect  effects  of  interest. 

The  second  study  involved  an  initial  assessment  of  the  capability  of  NCOT  to  support  real-time 
data  collection  using  intact  Ops  Room  teams  performing  tasks  of  picture  compilation  and  threat 
assessment.  The  outcome  of  this  assessment  led  to  a  third  report  which  provided  specific 
recommendations  for  software  enhancements  to  the  NCOT  facility  that  would  be  required  in  order 
to  allow  reliable  and  effective  data  collection  and  efficient  analysis.  One  limitation  of  the  NCOT 
environment  from  the  perspective  of  T&E  was  the  lack  of  an  adequate  capability  for  playback, 
review  and  analysis  of  team  behaviours  during  a  test  scenario.  Consequently,  the  fourth  report 
evaluated  COTS  products  to  provide  the  required  playback/analysis  capability  and  recommended 
that  it  be  integrated  into  the  NCOT  environment,  if  future  T&E  were  to  be  considered  in  this 
facility.  The  final  report  outlined  discussions  with  the  contractor  for  the  NCOT  software  on 
changes  that  would  be  required  in  the  functionality  of  the  existing  software  to  meet  the  needs  of 
future  T&E  trials. 


Page  30 


Final  COMDAT  Human  Factors  Report  Humansystems  Incorporated 


HUM  A  NS  YSTE  MS 


Discussion 

The  report  provides  a  good  set  of  initial  MOPs  that  could  potentially  be  used  to  evaluate  the  utility 
of  the  COMDAT  TD.  The  proposed  strategy  for  conducting  T&E  and  the  detailed  plan  provide  a 
useful  model  for  any  one  interested  in  conducting  human  centred  T&E.  It  identified  many  risks  that 
are  often  not  considered  until  too  late  in  the  process.  These  included 

•  the  ability  of  the  NCOT  facility  to  support  team-based  scenarios  and  capture  the  required 
T&E  data, 

•  the  availability  of  an  adequate  number  of  Navy  personnel  to  act  as  test  participants, 

•  the  exact  manner  in  which  the  TDP  will  be  implemented  and  how  it  can  be  interfaced  with 
the  test  environment, 

•  the  possibility  that  operational  realism  may  have  to  be  traded  against  the  requirement  to 
capture  reliable  and  valid  data,  and 

•  the  generalisability  of  the  findings  to  broader  mission  contexts. 

The  remaining  sections,  while  valid  at  the  time,  are  of  less  long  term  value.  In  the  end,  the 
capabilities  of  the  NCOT  facility  were  not  sufficient  to  support  the  requirements  for  full  scale  T&E 
trials.  With  the  additions  recommended  in  Sections  3  through  5  of  the  report,  it  could  be  usefully 
employed  for  evaluations  of  OMI  functionality  and  design  concepts  at  the  single  operator  level,  for 
which  the  minimal  overhead  and  logistics  requirements  would  provide  for  an  effective  and  efficient 
test  environment. 

4.4.3  Development  of  measures  of  performance  for  evaluating  the  COMDAT 
technology  demonstrator:  Potential  use  of  training  records  from  the 
Operations  Room  Team  Trainer  (ORTT). 


Abstract 

This  report  compiles  the  information  contained  in  three  Technical  Memoranda  pertaining  to 
the  development  of  Measures  of  Performance  (MOP)  for  Multi-Sensor  Data  Fusion  (MSDF) 
technology,  and  evaluates  logistics  and  the  utility  of  the  Operations  Room  Team  Trainer 
(ORTT)  to  gather  these  MOPs.  The  report  outlines  the  development  of  a  strategy  for  collection 
of  MOPs,  the  conduct  of  a  proof-of-concept  trial  at  ORTT  (with  attendant  identification  of 
potential  improvements  to  the  data  collection  strategy),  and  the  investigation  of  methods  to 
reduce  the  logistical  and  availability  problems  associated  with  obtaining  access  to  Subject 
Matter  Experts.  The  major  conclusion  of  the  report  was  to  recommend  against  the  conduct  of 
real  time  trials  dedicated  exclusively  to  the  collection  of  COMDAT  TD  relevant  MOPs. 
Instead,  it  was  proposed  that  MOPs  be  derived  from  the  archived  records  of  existing,  intact 
ship’s  teams  undergoing  evaluations  and  exercises  in  the  ORTT.  As  a  result,  a  full-scale, 
investigation  of  the  ability  of  the  archived  ORTT  training  records  data  to  provide  performance 
data  for  the  COMDAT  TDP  was  initiated. 
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Summary 

All  thinking  in  the  previous  studies  (Matthews,  Webb,  Stager,  and  Keeble  2001);  (Matthews, 

Webb,  Keeble,  and  Lamoureux  2004a)  about  the  methods  by  which  baseline  and  TD  enhanced 
MOPs  could  be  collected  had  centred  upon  the  conduct  of  live  trials  dedicated  specifically  to  T&E 
requirements.  However,  detailed  exploration  of  the  logistical  requirements  to  support  such  trials, 
indicated  that  such  an  approach  was  unlikely  to  be  implemented  given  the  availability  of  the 
ORTT,  supporting  logistical  personnel,  and  ships’  teams. 

The  first  section  of  this  report,  therefore,  outlines  an  alternative  method  for  gathering  MOP  data  to 
support  the  COMDAT  TDP.  This  method  involved  analysing  archived  performance  data  collected 
during  ship  work-ups  and  training,  therefore  requiring  less  logistical  coordination  and  control  over 
the  ORTT.  The  report  provides  an  initial  assessment  of  using  ORTT  training  records  as  a  potential 
source  of  COMDAT  relevant  MOPs,  based  on  the  observation  of  an  actual  team  training  exercise 
and  evaluation  of  the  real-time  data  record. 

Several  merits  to  the  proposed  approach  were  identified: 

•  No  additional  burden  on  the  Navy  for  T&E  trials  with  support  personnel  and  trial 
participants. 

•  Data  collected  from  a  wide  range  of  Navy  personnel,  thereby  improving  reliability  and 
generalisability  of  MOP  data  collected. 

•  Ongoing  use  of  the  ORTT  for  training  provides  many  opportunities  for  collecting  T&E 
data. 

•  A  large  ORTT  database  of  exercise  records  representing  Navy  personnel  operating  under 
quasi-operational  conditions  from  which  performance  data  can  be  extracted. 

•  Detailed  data  from  T&E  assessments  using  intact  teams  could  provide  important  insights  to 
the  Navy  about  team  performance  issues  not  readily  ascertained  when  the  team  is  assessed 
during  training. 

Using  pre-recorded  scenario  data  from  the  ORTT  was  also  seen  to  have  some  disadvantages: 

•  No  direct  T&E  control  over  the  trial,  for  example  to  vary  circumstances  in  a  systematic 
manner  or  to  inject  situational  awareness  "probes"  into  the  information  stream. 

•  Scenarios  could  not  be  repeated  precisely  because  training  exercise  controllers  introduce 
events  depending  upon  how  the  team  under  observation  was  performing. 

•  T&E  could  not  request  a  "SITREP"  to  assess  situational  awareness  of  any  team  member  at 
a  particular  point  in  time. 

Overall,  the  initial  assessment  indicated  that  there  was  merit  to  pursuing  this  approach  and 
recommended  that  a  “proof-of-concepf  ’  trial  be  conducted  to  make  a  final  determination  as  to 
whether  this  would  be  a  suitable  approach  for  efficiently  collecting  valid  and  reliable  data. 

The  second,  and  major,  section  of  the  report  outlines  the  conduct  and  outcome  of  the  proof-of- 
concept  trial.  For  COMDAT,  the  practical  goal  of  any  trial  was  to  evaluate  the  operational  impact 
of  MSDF  technology  on  critical  Ops  Room  tasks.  The  technology  impacted  most  directly  on  the 
picture  compilation  component  of  the  overall  detect-to-engage  process.  Therefore,  it  was 
recommended  that  the  technology  could  best  be  assessed  by  measuring  key  processes  in  the  detect- 
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to-resolve  cycle  using  appropriate  MOPs.  These  MOPs  were  expressed  in  quantitative  terms  such 
as  percent  accuracy,  percent  errors,  time  to  perform  certain  tasks,  total  time  spent  in  associated 
communications  and  percent  communication  errors. 

The  specific  goals  of  the  ORTT  POC  trial  were  as  follows: 

•  Observe  a  training  scenario  in  action  and  note  areas  of  interest  to  the  T&E  team  for 
potential  MOP  extraction. 

•  Assess  the  practicality  of  inserting  T&E  "tags"  into  the  scenario  record  during  scenario 
execution  to  assist  in  subsequent  MOP  analysis. 

•  Develop  a  methodology  for  analysis  and  recording  of  the  captured  data. 

•  Analyse  the  digital  video  and  audio  record  (using  ORTT  Debrief  software)  to  build  the 
ongoing  team  picture  and  communications  for  selected  events. 

•  Analyse  the  supplementary  video  and  audio  record  (using  a  standard  VCR)  to  build  the 
ongoing  team  picture  and  communications  for  selected  events. 

•  Derive  quantifiable  MOPs  from  the  previous  analyses. 

•  Determine  range  of  potential  MOPs  that  could  be  assessed  using  the  methodology. 

•  Make  recommendations  for  the  specific  subset  of  MOPs  that  would  be  most  suitable  for 
assessing  the  impact  of  MSDF  technology  in  future  trials. 

•  Comment  upon  the  effectiveness  and  efficiency  of  the  data  extraction  and  analysis 
methodology  and  its  implications  for  logistical  and  other  support  in  future  trials. 

Data  extraction 

For  the  purpose  of  the  pilot  study,  it  was  decided  that  relevant  data  would  be  extracted  from  the 
scenario  record  for  scenario  segments  that  focussed  on  air,  surface,  and  subsurface  picture  building. 
Scenario  playback,  and  therefore  data  extraction,  for  each  event  started  just  prior  to  initial 
registration  of  the  air,  surface  and  subsurface  contact  on  the  CCS.  Scenario  playback  and  data 
extraction  for  each  scenario  event  ended  at  the  point  of  resolution  or  at  the  end  of  threat 
engagement  for  the  air,  surface  and  subsurface  contact.  In  total,  it  took  approximately  12  hours  of 
initial  data  extraction  by  3  persons  to  capture  75  minutes  of  the  scenario. 

The  following  data,  extracted  from  video  (CCS  display)  and  audio  playback  of  scenario  events 
were  logged  directly  into  a  spreadsheet  by  the  T  &  E  team: 

•  Network  communications  including  the  specific  network  over  which  the  communication 
occurred  (e.g.  C  &  C,  TG  Command),  the  identity  of  the  person  sending  the 
communication,  the  identity  of  the  person  receiving  the  communication  and  the  content  of 
the  communication. 

•  Local  picture  information,  such  as  range  scale,  at  critical  points  in  the  scenario  playback 
(printouts  of  the  CCS  display  provided  this  information  as  well). 

•  Hooked  tracks  at  critical  points  in  the  scenario  playback. 

•  Time  of  occurrence  for  each  network  communication,  local  picture  change,  keystroke,  or 
significant  event  logged. 
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•  Additional  comments  noted  by  the  experimenter  for  future  reference.  For  example,  an  Ops 
Room  Team  member  might  have  been  on  a  different  network  channel  and  therefore  missed 
an  attempted  communication  from  another  Ops  Room  Team  member.  This  represents 
important  information  that  would  not  be  captured  by  the  above  data  extraction  methods. 

The  results  of  the  analysis  produced  sixty  potential  MOPs  across  air,  surface  and  sub-surface 
warfare  areas  that  could  be  extracted  from  the  records.  Of  these,  concrete  examples  and  data  were 
provided  for  nineteen  and  a  further  eleven  were  identified  as  being  feasible  to  extract,  given 
appropriate  scenario  circumstances  and/or  time  for  more  in-depth  analysis.  The  trial  clearly 
demonstrated  that  the  ORTT  data  record  provides  a  capability  to  capture  a  number  of  MOPs  that 
relate  to  the  critical  processes  of  the  detect-to-resolve  cycle  and  picture  compilation  and  associated 
communications.  The  report  provides  a  recommendation  for  the  preferred  subset  of  MOPs  that 
would  be  most  suitable  for  assessing  the  impact  of  MSDF. 

The  final  section  of  the  report,  which  dealt  with  an  unrelated  issue,  but  which  was  of  interest  to 
DRDC,  addresses  how  to  get  access  to  Subject  Matter  Experts  for  blocks  of  time  to  participate  in 
trials  or  to  meet  with  contractors  to  explain  operational  procedures  and  processes.  This  brief  report 
addresses  problems  in  obtaining  this  access  and  outlines  ways  to  lessen  demand  and  administrative 
burden  whilst  still  providing  the  required  SMEs  to  contractors. 

Discussion 

The  major  contribution  of  this  work  was  to  show  the  viability  of  analysing  data  from  existing 
ORTT  training  records  to  support  the  requirements  of  the  COMDAT  T&E  program  in  terms  of 
providing  baseline  metrics  for  existing  system  performance  with  respect  to  above  water  warfare 
tactical  picture  compilation.  The  primary  drawback  to  the  approach  is  that  it  is  currently  limited  to 
the  existing  CCS.  The  scenarios  are  designed  to  support  training  requirements  and  would  not 
necessarily  be  optimum  for  establishing  baseline  performance  of  tasks  relevant  to  specific 
Technology  Demonstrator  Projects. 
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5  Baseline  Performance  of  the  Halifax  Class 

Operations  Room 

5.1  Goals 

The  primary  purpose  of  this  work  thrust  was  to  provide  data  for  selected  MOPs  that  would  describe 
HIL  performance  on  the  existing  Halifax  Class  CCS  on  key  tasks  relevant  to  the  potential  future 
application  of  MSDF  technology.  The  MOPS  selected  were  derived  from  previous  report 
recommendations  as  outlined  in  Section  4  above.  The  approach  taken  was  based  directly  upon  the 
work  described  in  4.4.3  in  which  ORTT  records  captured  during  team  work-ups  were  subsequently 
analysed  and  metric  data  extracted. 

5.2  Responsible  parties 

All  of  this  work  was  contracted  to  Humansysfe/ws R  Incorporated. 

5.3  Reports  produced 

Matthews,  M.  L.,  Keeble,  A.  R.,  Bruyn  Martin,  L.,  and  Sartori,  J.  (2007).  Baseline  performance 
measures  of  the  detect-to-recognize  and  identify  process  in  anti-air  and  anti-surface 
warfare:  Analysis  of  Operations  Room  Team  Trainer  (ORTT)  training  records.  (DRDC 
Toronto  CR  2007-029).  Human systems  Incorporated,  Guelph,  ON. 

5.4  Summary  of  reports 

5.4.1  Baseline  performance  measures  of  the  detect  to  identify  process  in  anti-air 
and  anti-surface  warfare.  Analysis  of  Operations  Room  Team  Trainer  (ORTT) 
Training  Records. 


Abstract 

The  goal  of  this  project  was  to  create  a  baseline  data  set  that  would  represent  typical 
performance  of  the  processes  of  air  and  surface  contact  detection,  recognition  and  identification 
by  the  Operations  Room  team  of  HALIFAX  Class  frigates.  Archived  data  from  the  Operations 
Room  Team  Trainer  (ORTT)  Navy  training  records  were  analysed  to  yield  metric  data  for  a 
number  of  measures  of  performance  (MOPs)  for  critical  tasks  that  had  been  identified  from 
prior  Human  Factors  analyses.  The  detailed  analysis  of  the  resulting  data  set  is  provided  in  a 
classified  Annex.  Summaries  of  the  contact  types,  data  trends  and  broad  findings  are  reported 
below.  Limitations  in  the  data  and  their  utility  for  baseline  comparison  for  future  technology 
enhancements  are  discussed. 
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Summary 

As  part  of  the  HF  thrust  of  the  COMDAT  project,  it  was  determined  that  an  effort  should  be  made 
to  quantify  and  measure  any  performance  increments  that  might  accrue  through  future 
enhancements  to  the  Ops  Room  systems  of  the  Flalifax  Class  frigates  in  support  of  picture 
compilation.  To  support  this  effort,  several  HF  analyses  were  conducted  of  the  “detect-  to- 
identify”  process  and  MOPs  were  developed  that  focused  on  a  wide  range  of  operator  tasks  and 
operator  positions. 

Following  the  development  of  MOPs,  an  evaluation  was  performed  of  several  possible  Navy 
environments  that  would  support  the  requirements  for  conducting  T&E  trials  (Matthews  et  al. 

200 1 ;  Matthews  et  al.  2004a) 

As  a  result  of  this  assessment,  it  was  determined  that  the  logistical  requirements  for  implementing 
full  T&E  trials  were  too  difficult  to  surmount.  As  a  result,  it  was  decided  to  explore  the  possibility 
of  using  Navy  training  data  that  had  been  archived  in  the  ORTT  as  a  means  of  creating  baseline 
performance  metrics  for  critical  tasks,  against  which  future  technology  enhancements  could  be 
evaluated  (Matthews,  Webb,  Keeble,  and  Lamoureux  2004b). 

Proof  of  concept  trials  were  conducted  to  assess  the  feasibility  of  this  approach.  Following  the 
successful  outcome  of  these  trials,  a  more  sustained  effort  was  undertaken  to  capture  a  range  of 
performance  data  from  available,  archived  data  sets.  This  effort  is  the  subject  of  this  report. 

The  project  proceeded  in  two  phases:  initially,  only  air  warfare  was  analysed;  subsequently,  the 
analysis  focused  on  surface  warfare.  The  air  analysis  examined  four  data  sets  which  represented 
different  teams  participating  in  an  Operations  Team  Training  2  (OTT2)  assessment  (the  final 
assessment  that  “certifies”  that  they  are  fully  operational  and  ready).  Forty  four  air  contact  events 
(fifteen-assumed  friend;  one-suspect;  twenty  six-hostile  and  two-neutral)  were  analysed  from  the 
data  sets  and  task  time  data  were  extracted  for  the  following  primary  metrics: 

•  initiate  and  complete  resolve, 

•  first  and  subsequent  identification  (ID)  states,  and 

•  associate  Electronic  Support  Measures  (ESM). 

The  full  analysis  and  details  of  the  MOPs  data  are  provided  in  a  separate  classified  report.  It 
includes  data  on  task  performance  time  and  performance  errors.  It  was  concluded  that  the  majority 
of  the  errors  could  have  been  ameliorated  if  decision  aids  based  on  MSDF  were  available. 

In  the  case  of  surface  picture  compilation,  five  data  sets  were  analysed  from  ORTT  trials,  yielding 
a  total  of  ninety  seven  surface  contacts  that  were  recognized  and  identified.  Again,  the  results  of 
these  analyses  were  provided  in  a  separate  classified  report.  Surface  recognition  times  ranged 
widely  and  frequently  reflected  the  time  required  to  deploy  assets  closer  to  contacts  of  interest  in 
order  to  gain  further  recognition  data.  The  majority  of  contacts  were  recognized  to  “ possible )  high 
type ”,  followed  by  “prob(able)  class ”.  The  primary  information  used  for  recognition  assessments 
came  from  human  judgment  of  contact  behaviour  rather  than  from  sensor  data  in  itself.  Again, 
several  errors  were  identified  and  in  each  case,  the  availability  of  MSDF  decision  support  may 
have  mitigated  the  situation. 

The  report  concludes  with  a  discussion  on  the  reliability  and  generalisability  of  the  data  and  makes 
several  recommendations  for  future  work. 
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Discussion 

This  work  appears  to  have  met  the  requirement  to  provide  a  quantitative,  baseline  data  set  that 
represents  the  time  required  for  critical  tasks  in  air  and  surface  recognition  and  identification.  The 
data  accurately  represent  the  performance  of  the  current  system,  as  reflected  by  the  effort  of 
experienced  teams  working  with  complex  scenarios  that  are  designed  to  evaluate  and  “certify”  their 
operational  readiness.  There  was  sufficient  data  available  to  allow  many  of  the  important  time- 
based  metrics  to  be  assessed.  However,  in  order  to  reduce  some  of  the  variance  in  the  data,  some 
additional  data  sets  should  be  analysed. 

Limitations 

The  major  limitation  of  the  obtained  data  reflects  constraints  within  the  training  scenarios  with 
respect  to  the  number  of  simultaneous  contacts  that  a  team  is  expected  to  be  successful  in  dealing 
with,  and  the  rate  at  which  contacts  are  presented.  In  general  the  contact  level  and  rates  are  such 
that  a  competent  team,  working  efficiently,  does  not  fall  behind  in  the  recognition  and 
identification  process.  However,  it  has  been  frequently  suggested  that  MSDF  technology  may  be 
most  beneficial  in  those  circumstances  where  the  operating  environment  is  characterised  by  high 
air  and  surface  contact  rates,  with  a  heterogeneous  mix  of  contact  types.  Under  such  conditions, 
MSDF  would  increase  the  overall  system  performance  envelop  by  working  in  the  background  to 
process  all  contacts  and  compliment  the  work  of  the  team.  Therefore,  the  very  kinds  of  conditions 
(high  load  and  high  contact  rate)  where  one  would  like  to  assess  the  incremental  performance  due 
to  MSDF  are  not  represented  in  the  baseline  data  collected  in  the  ORTT.  The  report  recommends 
one  approach  to  solving  this  problem  by  using  the  baseline  performance  data  to  build  an  accurate 
task  network  model  to  simulate  the  identification  and  recognition  processes,  and  then  to  vary  the 
input. 
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Suggestions  for  improvement  to  the 
Operator  Machine  Interface  for  the  Halifax 
Class  CCS 


6.1  Goals 

The  goals  for  this  work  were  to 

•  Review  the  state  of  knowledge  with  respect  to  OMI  guidelines  relevant  to  the  CCS. 

•  Provide  system  designers  with  a  COMDAT  style  guide  that  would  contain  clear  guidelines 
on  the  design  of  screen  elements  and  interfaces  in  the  development  of  new  displays  for 
future  C2  systems  and  upgrades. 

•  Review  the  TD  for  compliance  with  the  style  guide. 

•  Conduct  an  analysis  of  methods  for  representing  contact  data  certainty  on  tactical  displays, 
obtain  operator  feedback  on  display  design  options,  and  recommend  a  preferred  OMI. 

•  Conduct  experiments  in  areas  where  additional  information  would  be  required  to  support 
the  development  of  the  COMDAT  OMI. 

6.2  Responsible  Parties 

The  first  4  goals  were  contracted  to  Unger  Campbell  and  Associates.  The  last  goal  was  completed 

by  DRDC  Toronto. 

6.3  Reports  produced 

Unger  Campbell,  G.  (2001).  Command  Decision  Aiding  Technology  (COMDAT)  Operator 

Machine  Interface  (OMI)  study:  Version  1.0.  (DCIEM  CR  2001-172).  Unger  Campbell 
and  Associates,  Delta,  B.C. 

Unger  Campbell,  G.  (2001).  Command  Decision  Aiding  Technology  (COMDAT)  Operator- 
Machine  Interface  (OMI)  style  guide:  Version  1.0.  (DCIEM  CR  2001-151).  Unger 
Campbell  and  Associates,  Delta,  B.C. 

Unger  Campbell,  G.  and  Baker,  K.  (2003).  Command  Decision  Aiding  Technology  (COMDAT) 
symbology  and  design  study.  (DRDC  Toronto  CR  2003-036).  Unger  Campbell  and 
Associates,  Langley,  BC. 

Unger  Campbell,  G.  (2004).  Command  Decision  Aiding  Technology  (COMDAT)  Technology 
Demonstrator  (TD)  Operator  Machine  Interface  (OMI)  Style  Guide  Compliance  and 
Enhancement.  (DRDC  Toronto  CR  2004-129).  Unger  Campbell  and  Associates, 
Lynnwood,  Washington,  USA. 
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Unger  Campbell,  G.  (2004).  Command  Decision  Aiding  Technology  (COMDAT)  Operator- 
Machine  Interface  (OMI)  style  guide:  Version  2.0.  Unger  Campbell  and  Associates, 
Langley,  B.C. 

McFadden,  S.M.,  Li,  A.  and  Trinh,  K.  (2007)  Representing  data  quality  on  naval  tactical  displays. 
(DRDC  Toronto  TR  2007-032).  Defence  R&D  Canada  -  Toronto,  Toronto,  ON. 

McFadden,  S.M.,  Jeon,  J.,  Li,  A.,  and  Minniti,  A.  (2007)  Evaluation  of  symbol  sets  for  naval 
tactical  displays  (DRDC  Toronto  TR  2007-046).  Defence  R&D  Canada  -  Toronto, 
Toronto,  ON. 


6.4  Summary  of  reports 

6.4.1  Command  Decision  Aiding  Technology  (COMDAT)  Operator  Machine 
Interface  (OMI)  study:  Version  1.0 


Abstract 

This  report  provides  supplemental  and  background  material  related  to  the  development  of  the 
COMDAT  OMI  Style  Guide  for  Navy  Command  and  Control  operations.  The  material  is 
presented  in  five  chapters.  Three  chapters  comprise  background  materials  as  follows:  the 
purpose  and  outline  of  the  COMDAT  OMI  Style  Guide,  a  literature  review,  and  a  discussion 
of  making  the  documents  available  on  the  World  Wide  Web.  The  remaining  two  chapters 
present  respectively  a  discussion  of  Naval  symbology  and  a  discussion  of  identified  Human 
Factors  and  Usability  issues  in  specific  command  and  control  functions  of  a  Halifax-Class 
Canadian  Patrol  Frigate  (CPF). 


Summary 

Purpose  of  the  style  guide 

The  purpose  of  creating  a  COMDAT  style  guide  was  to  provide  design  guidance  for  the  OMI  of  the 
COMDAT  technology  demonstrator.  The  intended  users  of  the  style  guide  were,  in  the  short  term, 
those  Defence  R&D  Canada  personnel  who  are  involved  in  the  development  of  the  COMDAT 
Technology  Demonstrator.  The  ultimate  users  of  the  technology  will  be  the  fleet  personnel, 
although  they  will  not  use  the  style  guide  per  se. 

In  the  long  term,  the  COMDAT  OMI  Style  Guide  should  apply  to  new  technology  that  will  be 
deployed  in  the  fleet.  Accordingly,  the  style  guide  was  designed  to  incorporate  each  of  the 
following  requirements: 

•  Adhere  to  current  forms  of  doing  business  (to  take  advantage  of  the  current  fleet 
knowledge  base). 

•  Address  current  HF  knowledge  and  practices  to  provide  enhancements  using  new 
information. 

•  Provide  for  growth  (to  permit  integration  of  new  technology  and  techniques). 
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Literature  review 

This  comprised  a  comprehensive  review  of  published  reports  on  a  wide  range  of  OMI  issues  for  C2 
from  a  variety  of  operational  domains.  The  majority  of  these  reports  were  published  in  the  last  ten 
years.  The  annotation  indicates  which  reports  provided  material  for  inclusion  into 
recommendations  for  the  style  guide. 

One  of  the  major  recommendations  to  emerge  from  the  style  guide  was  to  shift  from  a  traditional 
Open  Source  Format  (OSF)/Motif  style  to  a  Microsoft  (MS)  Window  style.  The  report  clearly 
identifies  those  aspects  of  the  latter  style  which  would  not  be  appropriate  for  application  in  a  Navy 
Halifax  Class  C2  context. 

Symbology 

The  recommendations  focus  on  the  symbology  needed  to  support  the  information  requirements  of 
the  back  row  Command  Team  of  a  Halifax  Class  Ops  Room.  They  cover  the  tactical  display 
symbology  for  these  operators  and  the  symbology  required  to  support  MSDF.  Issues  considered 
include  the  information  needs  of  the  team,  representation  of  MSDF  contacts  and  contact  certainty, 
and  representations  of  quantitative  and  qualitative  data. 

Identification  of  HF  and  usability  issues  in  the  Halifax  Class  Ops  Room. 

This  section  provides  information  to  developers  on  sources  of  information  for  implementing 
COMDAT  applications  or  for  upgrading  the  current  CCS.  Recommended  sources  are  observations 
conducted  during  a  sea  trial,  administered  questionnaires,  and  relevant  reports  produced  during  the 
COMDAT  TD  program. 

Publication  of  documents  on  the  World  Wide  Web  (WWW) 

Some  high  level  recommendations  are  provided  concerning  format,  links,  graphics,  and  usability. 

Discussion 

This  report  provides  a  useful  overview  of  the  process  used  in  compiling  the  final  OMI  Style  Guide 
and  provides  some  initial  recommendations.  The  literature  review  represents  a  timely  summary  of 
recent  relevant  work  on  OMI  design  for  a  variety  of  C2  application  environments.  Many  of  the 
recommendations  are  at  a  very  general  level  and  somewhat  preliminary.  They  become  refined  in 
later  reports. 

6.4.2  Command  Decision  Aiding  Technology  (COMDAT)  Symbology  and  Design 
Study 


Abstract 

The  report  includes  a  description  of  three  Human  Factors  efforts  carried  out  to  support  the 
upgrade  of  the  Halifax  Class  Command  and  Control  System  (CCS)  in  the  area  of  battlespace 
awareness.  The  first  section  is  comprised  of  a  report  on  an  evaluation  of  symbology 
developed  to  convey  the  certainty  of  Multi-Source  Data  Fusion  displays.  The  remaining 
sections  are  comprised  of  annotated  illustrations  of  COMDAT  Operator-Machine  Interface 
(OMI)  Style  Guide  compliant  displays. 
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Summary 

MSDF  certainty  symbology  evaluation 

A  fundamental  issue  surrounding  the  development  of  MSDF  technology  is  how  to  represent  fused 
information.  In  current  operations,  the  outputs  from  different  sources  are  presented  on  dedicated 
screens;  the  operators  have  information  about  the  certainty  of  the  data  because  of  the  information 
available  on  the  screen  on  which  it  appears.  If  contacts  from  a  variety  of  new  sources  were 
integrated  into  a  single  display  the  issue  arises  as  to  whether  there  is  a  need  for  symbology  that 
informs  the  operators  about  the  quality  of  the  information  and  if  so,  how  should  it  be  optimally 
presented.  Heretofore  there  have  been  several  attempts  to  develop  certainty  symbology  within  a 
variety  of  defence  applications  without  particular  success. 

Three  symbology  options  for  presenting  certainty  information  were  evaluated.  The  SMEs 
comprised  a  range  of  ranks  from  a  Halifax  Class  Ops  Room.  The  evaluation  involved  both 
performance  and  preference  measures  including  ratings  of  the  meaningfulness,  level  of  certainty 
represented,  and  perceived  usefulness  in  the  context  of  a  radar  tactical  display.  A  group  discussion 
of  the  candidates  and  other  alternatives  generated  some  design  concepts  that  were  considered  to  be 
desirable  for  representing  certainty  information. 

The  preferred  option  was  the  “slider”  or  “draining  bucket”  symbol  set  which  was  designed  to 
indicate  the  time  that  the  report  was  received.  In  this  symbol  set  the  more  full  the  “bucket” 
appeared  the  more  recent  the  report  and  hence  the  more  certain  the  contact. 

Of  interest,  the  SMEs  all  resisted  the  concept  that  the  MSDF  would  be  useful  to  them  at  the  CCS 
level.  In  the  discussion  group  that  followed  the  structured  evaluation,  100%  of  the  participants 
stated  that  they  did  not  find  the  concept  of  certainty  symbology  to  be  useful  for  the  CCS  except 
perhaps  to  help  with  situation  awareness  when  an  operator  first  comes  on  shift.  The  SMEs  indicted 
that  the  concept  would  be  more  useful  in  a  planning  display  rather  than  in  real  time  on  the  tactical 
display.  Arguments  made  by  the  SMEs  included  the  following: 

•  Any  contact  that  was  not  clear  would  be  examined  in  detail  by  the  operator  via  the  tabular 
displays.  The  SMEs  suggest  that  interpreting  the  certainty  measure  would  produce  a 
distraction  rather  than  help. 

•  Identification  is  not  an  issue  at  the  CCS  level.  The  identities  of  contacts  represented  on  the 
CCS  have  already  been  confirmed  elsewhere. 

•  Track  quality  is  already  identified  by  a  number.  These  SMEs  were  quite  clear  that  the  main 
concern  of  the  operators  is  whether  the  track  quality  is  good  enough;  the  quality  is  either 
good  enough  to  engage  or  it  is  not.  Levels  of  information  about  the  track  quality  when  it 
is  anything  less  than  “Good  to  Go”  are  less  critical.  It  is  not  clear  whether  the  less  critical 
information  is  used  at  all.  Additional  detail,  other  than  “Good  to  Go”  or  “Not  Good  to 
Go”  in  certainty  displays  is  perceived  by  the  operators  to  be  a  hindrance  rather  than  help 
during  a  real-time  CCS  task. 

Notwithstanding  these  comments,  which  largely  address  the  perceived  utility  of  such  a  display,  the 
report  makes  some  suggestions  for  an  enhanced  symbology  design  to  facilitate  recognition  and 
reduce  certainty.  Readers  should  refer  to  the  original  report  for  an  illustration  of  this. 
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OMI  design  illustrations 

Section  two  of  the  report  provides  specific  examples  of  symbology,  graphics,  and  OMI  elements 
that  would  amplify  the  first  version  of  the  COMDAT  OMI  Style  Guide.  These  enhancements  were 
subsequently  captured  into  the  final  version  of  the  Guide. 

CCS  look  and  feel  upgrade 

The  third  section  of  the  report  provides  extensive  examples  for  an  upgrade  to  the  CCS  OMI  to 
conform  to  the  proposed  style  guide,  largely  based  upon  the  report  authors’  HF  Engineering 
experience  and  knowledge  of  display  design  for  C2  environments.  These  were  later  incorporated 
into  the  final  version  of  the  style  guide  (Unger  Campbell  2004a). 

Discussion 

In  as  far  as  it  goes,  this  report  provides  some  valuable  information  concerning  how  track  certainty 
information  might  be  represented  in  a  future  MSDF  display.  Given  that  different  types  of  certainty 
were  considered  (time,  position,  identification,  source),  the  report  could  have  gone  further  in 
considering  the  saliency  of  these  different  types  of  certainty  for  operational  contexts  and 
considering  the  specific  design-representation  requirements  separately  for  each. 

The  report  recommends  the  use  of  the  slider  format  for  representing  certainty;  yet,  the  data  appear 
to  indicate  that  SMEs  preferred  the  dot  format. 

The  report  clearly  identifies  important  considerations  for  the  specific  design  of  MSDF  certainty 
symbology,  reflecting  concerns  of  clutter  and  unobtrusiveness  by  the  SMEs.  However,  some  of  the 
design  recommendations  may  be  too  vague  for  implementation  by  future  system  designers  for 

example:  “symbology  must  be  small  and  unobtrusive”,  “MSDF  symbols . must  be  as  compact  as 

possible”. 

In  general,  this  report  provides  a  useful  starting  point  for  considering  how  certainty  information 
may  be  best  represented  in  future  displays.  The  report  opens  the  door  to  many  questions  which 
will  need  to  be  addressed  before  specific  and  comprehensive  recommendations  can  be  made  with  a 
level  of  assurance.  Some  of  these  issues  are 

•  What  are  the  operation  decisions  where  representation  of  certainty  would  be  of  value? 

•  What  is  the  type  of  certainty  that  needs  to  be  considered  for  the  above? 

•  Are  the  needs  of  operators  to  reduce  uncertainty  about  information  different  at  differing 
phases  of  building  situation  awareness  in  a  tactical  environment? 

•  Are  there  preferred  methods  for  representing  the  level  of  certainty  for  different  types  of 
certainty  (time,  position,  source  etc)  -  or  should  a  common  method  be  used? 

Given  the  complexity  of  this  issue,  the  small  sample  set  and  limited  validation  by  the  operational 
community,  the  results  and  recommendations  of  this  report  are  probably  best  taken  as  suggestive 
and  pointing  to  the  requirement  for  further  research  and  analysis,  rather  than  providing  a  definitive 
answer  that  could  be  immediately  incorporated  into  requirements  for  future  upgrades  of  the  CCS. 
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6.4.3  Command  Decision  Aiding  Technology  (COMDAT)  Technology 

Demonstrator  (TD)  Operator  Machine  Interface  (OMI)  style  guide  compliance 
and  enhancement 


Abstract 

This  report  describes  the  results  of  a  review  of  the  prototype  COMDAT  TD  OMI  and 
describes  its  compliance  with  the  COMDAT  OMI  Style  Guide.  The  review  is  presented  in 
three  parts.  Part  one,  a  compliance  matrix,  identifies  over  120  design  elements  of  the 
COMDAT  TD  OMI  that  are  not  compliant  with  the  COMDAT  OMI  Style  Guide.  Each  of 
the  non-compliant  elements  is  described  in  the  compliance  matrix.  The  relevant  paragraph 
(and  paragraph  number)  from  the  COMDAT  OMI  Style  Guide  is  identified  for  each  non- 
compliant  design  element.  Part  two  identifies  24  new  items  that  will  be  added  to  the 
COMDAT  OMI  Style  Guide  as  a  result  of  the  review.  Part  three  covers  other  usability  issues, 
identified  during  the  review,  that  are  not  directly  related  to  style  guide  items.  The  review 
notes  include  specific  style  guidance  as  well  as  other  usability  observations  from  the 
COMDAT  TD  review. 


Summary 

Background 

As  part  of  the  TD  development,  new  design  concepts  were  being  developed  and  implemented  by 
Lockheed-Martin  to  (a)  incoiporate  the  output  from  MSDF  processes  into  the  current  CCS  tactical 
picture  and  (b)  integrate  the  wide  area  picture  as  provided  by  GCCS-M  and  Link- 1 1  with  the 
Ownship  tactical  picture.  The  new  functionality  allowed  operators  to  access  and  manipulate  this 
additional  information.  Thus,  the  initial  design  of  the  OMI  for  the  COMDAT  TD  deviates 
substantially  from  the  current  CCS  tactical  display. 

An  initial  review  of  the  design  concepts  developed  during  the  second  build  of  the  COMDAT  TD, 
and  the  OMI  itself,  showed  that  they  were  not  completely  consistent  with  the  COMDAT  OMI  Style 
Guide  or  with  good  HF  practice.  Therefore,  as  a  precursor  to  specifying  additional  enhancements 
to  the  OMI,  it  was  necessary  to  conduct  a  formal  review  of  the  current  COMDAT  TD  OMI. 

The  intent  of  the  review  was  to  identify  inconsistencies  between  the  COMDAT  TD  OMI  and  the 
COMDAT  OMI  Style  Guide  so  that  these  inconsistencies  could  be  addressed.  The  ultimate  goal 
was  to  recommend  an  interface  that  was  consistent  with  the  COMDAT  OMI  Style  Guide  and  with 
good  HF  practice. 

COMDAT  OMI  review  process 

The  author  conducted  an  initial  walkthrough  of  the  COMDAT  TD  OMI,  followed  by  a  detailed 
analysis  of  the  compliance  of  the  OMI  with  Version  1  of  the  OMI  style  guide.  This  review  appears 
to  have  been  conducted  by  the  report  author  alone.  The  two  outputs  of  the  review  process  were  a 
compliance  matrix  and  some  general  observations. 

Compliance  with  the  style  guide 

Specific  elements  of  the  COMDAT  TD  design  that  were  not  compliant  with  the  COMDAT  OMI 
Style  Guide  were  identified  and  were  recorded  in  a  compliance  matrix.  It  identified  123  areas 
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where  the  COMDAT  TD  could  benefit  by  re-design  so  as  to  be  compliant  with  the  COMDAT  OMI 
Style  Guide. 

Since,  the  COMDAT  TD  development  was  part  of  the  Naval  Tactical  Display  (NTD),  and  was 
limited  in  scope  by  requirements  to  retain  some  aspects  of  the  NTD  displays,  the  compliance 
matrix  took  into  account  these  limitations  in  scope  but  was  not  restricted  to  those  design  features 
that  were  solely  part  of  the  new  TD. 

Only  those  elements  of  the  style  guide  relevant  to  the  current  build  of  the  COMDAT  TD  OMI  were 
included  in  the  compliance  matrix. 

The  non-compliant  components  were  found  to  cover  a  wide  range  of  different  OMI  issues,  which 
are  summarised  below: 

•  format  of  displays  (fonts,  symbology,  organisation,  appearance,  formatting  of  text,  colour 
coding), 

•  display  navigation, 

•  windows  management, 

•  match  to  user  requirements  and  needs, 

•  input  controls  and  data  entry, 

•  error  handling, 

•  consistency  with  MS  Windows, 

•  overlays  and  tools, 

•  hooking  tracks, 

•  track  coding,  and 

•  representation  of  certainty 
Revisions  to  the  style  guide 

As  a  result  of  the  review,  a  number  of  issues  arose  that  required  either  the  addition  or  removal  of 
items  from  the  first  version  of  the  style  guide.  Such  changes  were  then  incoiporated  into  what 
became  the  final  version. 

Author’s  general  impressions  of  the  OMI 

The  primary  and  overwhelming  impression  was  that  the  controls  and  displays  were  too  complex  for 
operational  use.  The  impression  was  of  a  system  that  was  designed  to  reflect  and  provide  access  to 
the  underlying  technology,  rather  than  to  enhance  the  ability  of  the  operators  to  complete  their 
missions  successfully. 

The  MSDF  displays  appeared  to  reflect  the  research  community’s  requirements  rather  than  the 
operator’s  requirements.  While  the  displays  and  controls  did  permit  the  tasks  to  be  completed,  the 
designs  were  not  focused  on  the  operators’  tasks  or  missions.  In  an  operational  environment,  the 
effectiveness  of  the  MSDF  would  benefit  significantly,  and  be  very  useful  to  the  operational 
effectiveness  of  the  ship,  with  modifications  to  the  display  concept  that  were  consistent  with  good 
HF  principles  within  the  operational  context. 
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Discussion 

This  report  provides  a  very  thorough  review  of  the  prototype  TD  OMI  and  makes  a  number  of 
critical  and  useful  recommendations  to  enhance  the  utility  and  usability  of  the  system  functionality. 

For  system  developers,  it  would  have  been  useful  to  provide  some  indication  within  the  compliance 
matrix  of  the  severity  of  the  consequences  of  non-compliance,  thereby  allowing  some  prioritisation 
of  remedies.  (For  example,  “minor”  issues  such  as  misaligned  text  or  tab  fields  are  not 
differentiated  from  more  operationally  significant  problems  such  as  difficult  to  navigate  screens). 

In  particular  some  of  the  recommendations  concerning  issues  that  impact  upon  critical  tasks  and 
operations  are  not  given  sufficient  weight  or  highlighting  in  the  report.  These  issues  include  track 
numbering  by  MSDF,  which  has  significant  operational  consequences  for  track  management  and 
contact  recognition,  representation  of  certainty  in  the  MSDF  contact,  operator  access  to  underlying 
sensor  data  for  MSDF  tracks,  and  access  to  the  MSDF  tactical  display  screen. 

The  following  limitations  and  points  should  be  considered  when  assessing  the  value  of  the 
information  provided: 

•  The  recommendations  appear  to  be  the  opinion  of  one  F1F  qualified  person  alone.  They 
have  not  been  validated  by  a  second  F1F  SME  and,  in  particular,  recommendations  that 
reflect  operational  requirements  do  not  appear  to  have  been  validated  by  Navy  SMEs. 

•  Statements  such  as  “text  is  not  legible”  are  not  informative  for  system  redesign  purposes. 

It  would  have  been  better  to  have  expressed  this  along  the  lines  of  “text  height  was  . .  ,x 
min  at  the  operator  preferred  viewing  distance  of . .  .cm;  this  does  not  conform  to  guideline 
..,  which  states  that  text  must  have  a  minimum  visual  angle  of . . ..  Therefore  increase  the 
height  of  this  text  to  . .  .mm”.  Similar  comments  apply  to  statements  such  as  “The  hooked 
track  should  itself  become  brighter  or  larger  in  order  to  confirm  that  it  is  indeed  the  hooked 
track.” 

6.4.4  Command  Decision  Aiding  Technology  (COMDAT)  Operator-  Machine 
Interface  (OMI)  style  guide:  Version  1.0. 

Command  Decision  Aiding  Technology  (COMDAT)  Operator-  Machine 
Interface  (OMI)  style  guide:  Version  2.0. 


Abstract 

This  document  provides  an  Operator-Machine  Interface  (OMI)  style  guide  for  the  Canadian 
Command  Decision  Aiding  Technology  (COMDAT)  project.  This  COMDAT  OMI  Style 
Guide  has  been  created  to  support  development  in  Command  Decision  Aids  for  Halifax-Class 
Canadian  Patrol  Frigates  (CPFs).  The  objective  is  to  provide  guidance  to  create  a  common 
look  and  feel  that  is  compatible  with  existing  systems,  yet  accommodates  new  developments 
and  knowledge. 


Summary 

The  initial  version  of  the  style  guide  was  published  as  a  DRDC  report  (Unger  Campbell  2001a). 
Under  later  contracts,  it  was  implemented  as  an  HTML  document  (Unger  Campbell  2004a)  and 
upgraded  to  include  the  OMI  design  illustration  discussed  in  Section  6.4.2  (Unger  Campbell  and 
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Baker  2003)  and  the  modifications  and  additions  arising  out  of  the  evaluation  of  the  COMDAT  TD 
(Unger  Campbell  2004b).  The  summary  and  discussion  apply  to  the  final  version  of  the  style  guide. 

Introduction 

The  COMDAT  OMI  Style  Guide  provides  design  guidance,  design  rules  and  a  framework  for  the 
OMI  design  rule  development  process.  There  are  4  overall  objectives  of  the  style  guide: 

•  compatibility  with  the  current  Halifax  Class  CCS, 

•  employment  of  an  interface  style  with  which  the  operators  are  familiar, 

•  compatibility  with  existing  military  OMI  style  guides,  and 

•  ability  to  accommodate  new  features  and  functionality. 

It  was  hoped  that  this  guide  would  lead  to  OMI’s  that  are  consistent,  interoperable  and  usable. 

A  major  change  from  the  existing  CCS  is  a  recommendation  to  adopt  the  Microsoft  Windows  style 
as  a  template  for  the  look,  feel  and  functionality  of  new  systems. 

Design  filters 

The  expected  end-users  of  the  style  guide  will  be  developers  of  technology  for  Maritime  CCS 
operational  personnel.  Developers  need  to  design  for  the  lowest  anticipated  operator  skill  set, 
while  ensuring  support  for  high  level  operators.  The  physical  environment  of  CCS  operators  (i.e. 
shipboard  Ops  Room)  must  also  be  taken  into  account  in  the  design. 

Source  documents 

Thirteen  source  documents  are  cited  throughout  the  style  guide. 

Input  devices 

Input  devices  for  the  COMDAT  CCS  include,  at  minimum,  a  keyboard  and  a  pointing  device. 
Touch  screens  are  suitable  for  infrequent  operator  tasks,  but  voice-activated  controls  are  not 
suitable  (at  this  time)  for  Maritime  C2  function.  Specific  pointer  shapes  and  function  are  outlined, 
along  with  expected  functioning  of  the  input  devices. 

Windows 

There  are  3  different  types  of  windows  outlined:  system  (or  root)  windows,  primary  windows  and 
secondary  windows.  Characteristics  of  each  type  of  window  are  described  with  specific  attention 
paid  to  ‘status  bar’  and  other  information  bars  that  are  required  on  each  window. 

Design  guidance 

A  variety  of  issues  that  designers  would  need  to  address  are  discussed;  notably: 

•  consistency, 

•  arranging  information, 

•  three  levels  of  window  modality, 

•  widget  selection, 

•  coding  critical  information,  and 

•  window  navigation  and  selection. 
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The  report  provides  detail  on  how  input  devices  attain  window  focus  and  control  objects, 
specifically  using  a  pointer  device  and  keyboard.  The  thorough  discussion  of  object  control 
outlines  four  methods  of  transferring  objects,  feedback  operations,  error  detection,  and  undo 
capability.  Additionally,  a  chart  provides  specific  maximum  system  response  times  for  certain 
system  inputs  (e.g.  response  time  for  the  appearance  of  a  character  after  a  key  is  pressed  should  be 
less  than  0.2  seconds). 

Controls 

The  ‘Controls’  section  of  the  report  lists  characteristics  and  actions  that  the  controls  in  any  CCS 
system  should  have.  Different  rules  are  outlined  between  controls  associated  with  very  frequent 
and/or  safety  critical  tasks  (e.g.  hooking  tracks  or  changing  the  range  in  the  tactical  display)  and 
controls  for  more  generic  tasks.  Also,  design  rules  are  outlined  for  the  currently  used  Quick 
Access  Buttons  to  increase  their  visual  affordance2.  Finally,  each  of  the  different  types  of  controls 
(e.g.  push  buttons,  radio  buttons,  and  checkboxes)  is  discussed  with  a  view  to  providing  guidance 
on  streamlining  their  implementation. 

Menus 

Menu  hierarchy  structures  are  reviewed,  including  the  advantages  (e.g.  decreased  cognitive 
requirements)  and  disadvantages  (e.g.  slower  for  experienced  operators);  the  elements  and 
operations  of  menus  are  thoroughly  laid  out. 

Window  states 

The  report  notes  that  windows  along  with  their  associated  menus  and  title  bars  must  have 
consistent  functionality  to  ensure  the  operators  do  not  waste  time  searching  for  the  information 
they  need.  A  detailed  description  of  windows  is  included. 

Security  and  simulation 

Security  login  and  logout  procedures  are  specified.  Simulation  data  must  be  easily  distinguishable 
from  real  data,  and  during  any  simulation  real  data  must  be  protected. 

Data  display  and  entry 

Rules  for  data  entry  include  navigation,  pointer  activity,  and  data  manipulation  for  both  alpha  and 
numeric  characters.  The  data  display  rules  similarly  define  use  and  organization  of  both  alpha  and 
numeric  characters.  Special  attention  is  paid  to  error  correction. 

Special  functions  and  formats 

Except  for  critical  situations,  the  report  recommends  that  audible  alerts  not  be  implemented. 
Date/time  and  latitude/longitude  need  to  be  consistently  displayed  and  messages  need  to  be 
consistently  handled.  Additionally,  the  spell  check,  printing,  and  imagery  manipulation 
functionality  are  described. 

Graph  display 

The  characteristics  of  line  graphs,  bar  graphs  and  histograms  are  outlined  to  expedite  efficiency  in 
readability  and  comparisons  across  graphs. 


2  Within  the  HF  community,  affordance  is  defined  as  those  action  possibilities  which  are  readily  perceivable 
by  an  actor. 
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Tactical  graphics 

The  usability  of  tactical  graphics  is  central  to  operator  effectiveness.  Issues  that  are  specifically 
discussed  include  optimizing  the  use  of  colour,  shapes,  and  size  for  differentiability. 

Maps  and  situation  displays 

The  usability  of  the  tactical  display  is  also  central  to  operator  effectiveness.  General  tools  to 
manipulate  the  display  and  characteristics  of  the  display  are  discussed.  The  specific  CCS 
functioning  of  hooking  is  also  included. 

Colour 

The  final  section  of  the  report  outlines  general  rules  for  the  use  of  colour.  The  report  provides 
detailed  guidelines  on  how  colour  is  to  be  implemented  on  displays  depending  on  the  working 
environments  ambient  lighting  (e.g.  dark  adaptation  versus  normal  lighting).  Readers  of  this 
information  will  require  familiarity  with  colour  space  specification  metrics  in  order  to  fully 
comprehend  the  recommendations. 

Annex  1:  Highlights  of  the  COMDAT  OMI  Style  Guide 

Annex  1  includes  a  high  level  overview  of  the  style  guide  and  draws  particular  attention  to  map 
controls,  display  area  layout,  operator  actions  and  the  COMDAT  status  bar. 

Annex  2:  Illustrations  of  a  style  guide  compliant  CCS 

Annex  2  provides  a  static  re-design  of  the  current  Halifax  Class  CCS  functionality  and  layout  along 
with  an  explanation  of  the  changes  that  have  been  implemented.  The  upgraded  design  was  created 
to  illustrate  the  application  of  the  COMDAT  OMI  Style  Guide  to  the  current  CCS. 

Discussion 

The  style  guide  provides  a  large  amount  of  detailed  information  to  system  designers  looking  to 
implement  evolutionary  technology  for  Maritime  CCS.  The  majority  of  the  report  describes  in 
considerable  detail  Microsoft  Windows  styles  features  and  operation.  The  few  sections  that 
address  operationally  specific  support  requirements,  stress  the  continued  reliance  on  the  current 
display  functionality  in  order  to  minimize  transfer  effects  and  training  costs,  which,  as  a  result,  may 
limit  more  drastic  changes  in  design. 

The  HTML  implementation  of  the  style  guide  provides  a  highly  usable  approach  for  locating 
information. 

The  body  of  the  style  guide  provides  very  little  higher  level  information  concerning  overall  flow, 
style,  and  usability  of  the  CCS  system.  While  Annex  1  does  contain  this  type  of  information,  a 
greater  number  of  high  level  summaries  in  each  section  would  be  beneficial  for  those  looking  to 
contribute  to  designs  of  the  system  but  who  are  not  responsible  for  implementation  (and  therefore 
do  not  require  the  detailed  information).  More  pictures  and  figures  to  enhance  the 
comprehensibility  of  some  design  concepts  would  have  been  useful. 

A  logical  next  step  is  to  create  a  software  package  with  frames  and  widgets  consistent  with 
Microsoft  Windows  Styles  in  a  programming  language  that  works  with  the  current  CCS  system. 
This  would  greatly  ease  the  required  work  for  designers  who  are  looking  to  test  and  implement 
additions/changes  to  the  current  system.  The  software  package  could  be  similar  to  how  ‘Microsoft 
Visual  Basic’  automatically  creates  certain  functionality  (e.g.  by  default,  every  window  frame  has  3 
buttons  in  the  upper  left  comer  for  minimizing,  maximizing,  and  closing)  and  provides  global 
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functionality  to  be  similar  across  programs  (e.g.  access  to  directories  and  folders  through  input 
windows  like  ‘Save  As’  and  ‘Open’).  This  type  of  package  would  ensure  a  faster  implementation 
of  Windows-like  designs  by  programmers  who  would  otherwise  delay  programming  the  details 
associated  with  Windows  styles  in  order  to  expediently  implement  their  additions/changes. 

Limitations 

One  of  the  major  areas  not  addressed  in  the  style  guide  concerns  how  to  provide  functionality  at 
each  work  station  that  addresses  the  specific  operational  needs  of  the  user  for  different  phases  of 
operations  (e.g.  picture  compilation,  threat  assessment,  response  and  response  management). 
While  this  may  not  have  been  the  mandate  for  the  contracted  work,  there  will  be  a  need  for  future 
systems  to  provide  context-  relevant,  operational  information  tuned  to  the  needs  of  different  team 
members.  That  is,  while  the  style  guide  provides  detailed  guidance  for  the  display  of  data,  it  does 
not  address  the  higher  cognitive  requirements  of  the  team  in  terms  of  information  display. 

6.4.5  Representing  data  quality  on  naval  tactical  displays 


Abstract 

Multi- Source  Data  Fusion  (MSDF),  as  developed  under  COMDAT,  provides  an  assessment  of 
the  reliability  of  the  estimate  of  the  fused  track’s  attributes  and  position.  Part  of  the  human 
factors  work  under  COMDAT  has  been  to  investigate  methods  of  representing  the  quality  of  the 
MSDF-generated  tracks  to  the  operator.  The  research  reported  in  this  paper  is  concerned  with 
the  potential  impact  of  different  representations  of  data  quality  or  uncertainty  on  the  visibility  of 
tactical  symbols  and  the  intuitiveness  of  the  different  representations.  Three  methods  of 
representing  data  quality  were  investigated:  a  variably  filled  bar  presented  beside  the  tactical 
symbols,  different  diameter  rings  that  encompassed  the  tactical  symbols,  and  varying  the 
saturation  of  the  tactical  symbols.  In  Experiment  1,  a  visual  search  task  was  used  to  compare  the 
accuracy  and  speed  with  which  participants  could  locate  multiple  instances  of  each  of  the 
tactical  symbols  without  any  representation  of  data  quality  and  when  each  of  the  three  methods 
tested  were  added.  Experiment  2  examined  the  ability  of  participants  to  quickly  and  consistently 
interpret  the  quality  of  the  data,  represented  by  the  tactical  symbol,  using  the  three  different 
methods.  The  results  indicated  that  the  bar  interfered  least  with  people’s  ability  to  locate  the 
tactical  symbols,  but  the  saturation  method  was  most  consistently  interpreted.  A  second  set  of 
experiments  looked  at  applying  the  saturation  coding  to  the  bars  and  rings  instead  of  the  tactical 
symbol.  Redundant  saturation  coding  of  the  bars  and  rings  had  no  effect  on  tactical  symbol 
visibility  and  did  not  improve  the  consistency  of  interpretation  of  data  quality.  The  basic 
recommendation  was  that  a  small  independent  symbol,  such  as  the  bar,  was  the  preferred 
method  for  representing  data  quality.  However,  operators  should  have  the  option  of  turning  it 
off  if  the  display  appears  too  cluttered.  If  a  different  symbol  shape  is  chosen,  its  intuitiveness 
should  be  assessed  prior  to  implementation.  Further  research  is  required  to  improve  our 
understanding  of  the  number  of  levels  of  data  quality  operators  can  use  effectively. 


Summary 

This  report  examines  how  to  represent  the  data  quality  of  MSDF  tracks  to  the  operator.  This  is  an 
important  issue,  since  knowledge  of  the  reliability  of  the  track  data  could  be  of  potential  benefit  to 

Page  50  Final  COMDAT  Human  Factors  Report  Hnmansystems  Incorporated 


HUM  A  NS  YSTE  MS 


operators  concerning  action  to  take  on  the  contact.  Also,  since  operators  raised  concern  about 
display  clutter  in  the  Unger  Campbell  study,  design  approaches  needed  to  be  found  to  overcome 
this,  while  still  allowing  for  the  representation  of  data  quality. 

The  report  reviews  some  of  the  existing  methods  for  representing  information  certainty  and 
concludes  that  the  evidence  suggests  that  symbology  which  incorporates  certainty  information,  or 
putting  certainty  information  directly  on  a  tactical  display,  appear  to  be  a  better  option  than 
standalone  certainty  displays.  However,  the  implications  of  putting  such  information  directly  on 
displays  raises  the  issue  of  clutter  and  this  does  not  appear  to  have  been  empirically  investigated. 
Further,  the  research  on  how  to  represent  different  levels  of  certainty  has  not  addressed  the 
question  of  how  a  specific  level  of  certainty  can  be  mapped  onto  symbology  in  order  that  an 
operator  can  understand  and  inteipret  it  unambiguously. 

The  report  provides  a  good  summary  overview  of  recent  work  on  the  representation  of  certainty 
and  notes  the  deficiencies  in  many  studies  to  date. 

As  a  consequence  of  a  lack  of  definitive  answers  in  the  literature,  the  study  had  the  goal  of 
examining  graphical  methods  and  specific  symbology  as  a  way  to  represent  recognisable  certainty 
levels  by  measuring  performance  on  criterion  tasks. 

In  the  first  two  experiments,  the  stimuli  were  based  on  the  Naval  Tactical  Data  Set  (NTDS)  and 
equivalent  symbols  from  Mil-Std  2525B;  each  symbol  was  annotated  with  a  bar  (to  the  right  of  the 
symbol)  or  ring  (surrounding  the  symbol)  or  the  saturation  was  varied.  Three  levels  of  data  quality 
were  indicated  by  the  proportion  of  the  bar  filled  in,  the  size  of  the  ring,  or  the  degree  of  saturation. 

Operator  tasks  were  to  either  count  the  number  of  target  symbols  on  the  display  as  quickly  and 
accurately  as  possible  or  search  for  a  specific  symbol  and  indicate  the  data  quality. 

The  results  of  the  counting  task  were  somewhat  complex  but  generally  showed  that  performance 
with  the  saturation  coding  was  generally  poorer  than  with  the  remaining  methods  and  that 
performance  with  the  ring  coding  was  poorer  than  the  control  condition.  For  the  classification  task, 
only  the  saturation  coding  condition  showed  a  consistent  use  of  the  5  point  rating  scale  by  all 
subjects;  however,  the  bar  coding  showed  a  trend  for  significantly  different  ratings  for  each  level. 

The  conclusion  from  the  first  two  experiments  was  that  none  of  the  methods  were  entirely 
satisfactory  for  representing  data  quality.  The  saturation  method  resulted  in  the  most  consistent 
response,  but  it  severely  impacted  the  response  time  and  accuracy  with  which  the  tactical  symbols 
were  located.  The  bar  method  had  the  least  impact  on  response  time  and  accuracy,  but  there  was 
some  inconsistency  in  the  participants’  interpretation  of  the  different  levels  of  fill.  The  range  rings 
were  the  least  successful.  They  were  not  consistently  inteipreted  across  participants  and  accuracy 
and  response  time  for  counting  the  symbols  was  significantly  poorer. 

Accordingly,  a  second  pair  of  experiments  was  conducted  in  which  saturation  (the  preferred  coding 
method)  was  combined  with  a  secondary  symbol  (bar  or  ring);  it  was  predicted  that  performance 
would  be  superior  to  a  monochrome  condition.  The  second  experiment  was  conducted  with  the 
NTDS  symbology. 

The  results  for  the  counting  task  were  somewhat  similar  to  the  first  experiment,  but  there  was  no 
significant  performance  improvement  resulting  from  the  redundant  coding  of  data  quality.  For  the 
classification  task,  the  bar  method  was  again  found  to  be  superior  to  the  ring  method  for  coding 
quality. 
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Overall,  the  results  showed  that  the  addition  of  a  bar  representation  had  a  relatively  small  impact  on 
response  time  for  locating  specific  symbols.  Paradoxically,  while  saturation  coding  of  data  quality 
was  more  readily  interpreted,  it  resulted  in  symbols  that  took  longer  to  count. 

In  conclusion,  the  authors  raise  the  important  point  that  certainty  in  tactical  displays  may  be 
manifested  in  more  than  one  dimension  of  a  contact  (e.g.  track  location,  time  lateness,  platform  and 
ID)  and  that  it  would  be  unworkable  to  try  and  represent  certainty  for  each  of  these  because  of  the 
impact  on  display  clutter.  Therefore,  it  was  concluded  that  a  symbol  should  provide  only  general 
information  about  the  overall  quality  of  data  in  order  to  avoid  potential  clutter  and  increased 
operator  information  load. 

Discussion 

This  report  addresses  an  important  issue  concerning  how  to  represent  data  quality  to  operators 
using  symbology  that  is  readily  interpreted  but  does  not  clutter  displays  and  allows  symbols  to  be 
quickly  located.  The  experiments  conducted  provide  some  useful  information  concerning  a 
potential  way  ahead  for  such  representations  and  clearly  show  what  types  of  methods  are  less 
successful.  In  particular,  the  experiments  show  the  difficulty  of  trying  to  incorporate  iconic 
information  about  a  contact  into  the  contact  symbology.  The  results,  however,  are  suggestive  of 
future  directions  rather  than  being  able  to  provide  definitive,  immediate  solutions  that  would 
translate  directly  into  either  guidelines  or  deployable  design  concepts  (which  was  not  the  intent  of 
the  study).  The  results  clearly  indicate  that  more  empirical  work  is  necessary  before  the  important 
variables  are  identified  that  influence  how  to  code  for  data  quality  in  tactical  displays. 

Limitations 

The  report  does  not  discuss  the  issue  of  how  to  trade-off  detailed  information  on  data  quality  with 
issues  of  clutter,  particularly  as  operators  range  out. 
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6.4.6  Evaluation  of  symbol  sets  for  naval  tactical  displays 


Abstract 

The  four  experiments  reported  in  this  paper  were  conducted  in  support  of  the  COMmand 
Decision  Aiding  Technology  (COMDAT)  (1  lbg)  Technology  Demonstrator  Project  (TDP) 
and  the  Halifax  Class  Modernization  Command  and  Control  System  (HMCCS)  programme. 
The  first  experiment  assessed  the  relative  visibility  of  the  basic  Naval  Tactical  Data  System 
(NTDS)  and  MIL-STD-2525B  (2525B)  tactical  symbols.  Performance  with  the  colour-coded 
versions  of  the  two  symbol  sets  was  not  significantly  different.  However,  the  air  and 
subsurface  symbols  were  less  discriminable  than  the  surface  symbols.  Recommendations  for 
improving  the  discrimination  of  the  different  warfare  area  symbols  are  included. 

One  potential  advantage  of  the  2525B  symbols  is  the  possibility  of  adding  additional 
information  about  the  track  platform  to  the  basic  symbol  shape.  The  remaining  experiments 
assessed  the  visibility  of  the  basic  symbols  with  iconic  information  added  and  the  visibility  of 
the  icons  themselves.  Adding  iconic  information  did  not  have  a  large  effect  on  the  efficiency 
with  which  the  basic  symbols  were  located  except  when  the  icon  shape  replaced  the  symbol 
shape.  Performance  in  locating  and  recognizing  individual  icons  depended  on  their 
complexity  and  their  uniqueness.  It  was  recommended  that  only  a  small  number  of  highly 
discriminable  icons  be  used  at  any  one  time.  The  use  of  icons  without  the  symbol  frame 
should  be  restricted  to  non  tactical  information  and  such  icons  should  probably  not  be  colour 
coded.  Further  research  is  required  to  determine  how  to  implement  these  recommendations. 


Summary 

The  laboratory  experiments  reported  in  this  paper  were  all  designed  to  address  issues  concerning 
the  development  of  improved  OMI  guidelines  to  support  naval  tactical  picture  compilation.  A 
critical  first  step  in  this  process  is  the  ability  of  operators  to  quickly  and  accurately  discriminate 
contact  symbols  of  interest  from  among  other  contacts  in  the  tactical  picture,  as  well  as 
discriminate  contact  symbology  from  complex  backgrounds  that  may  contain  a  map  or  other 
graphical  information  (e.g.  a  water  space  management  box).  Therefore,  the  design  of  symbols  to 
support  this  requirement  becomes  an  important  issue  that  cannot  be  left  to  software  designers; 
hence,  guidelines  have  been  proposed  to  constrain  and  inform  the  design  implementation. 

The  experiments  reported  provide  a  much  needed  empirical  evaluation  of  the  effectiveness  of  two 
existing  guidelines  in  promoting  symbology  that  supports  the  tactical  requirements  of  operators. 
The  two  guidelines  are  the  NTDS  symbol  set  currently  used  on  tactical  displays  in  the  Halifax 
Class  frigates  and  MIL-STD-2525B  (2525).  The  results  of  these  experiments  will  be  used  to 
provide  recommendation  to  the  Navy  on  the  selection  of  suitable  symbology  for  naval  tactical 
displays. 

The  introduction  section  of  the  paper  provides  a  fairly  detailed  overview  of  the  historical 
background  research  and  analysis  in  the  development  of  military  symbology  in  the  last  two 
decades.  This  is  a  period  in  which  there  have  been  significant  technological  advances  in  coloured 
display  capabilities.  There  was  clearly  a  need  to  re-evaluate  the  NTDS  symbols  against  other 
candidates  to  determine  whether  the  design  concepts  were  still  valid,  since  they  were  developed  in 
the  early  days  of  electronic  display  in  operations  centres.  Over  the  same  period,  there  were 
considerations  that  symbology  could  be  augmented  through  the  additions  of  iconic  information  to 
complement  the  shape  and  colour  coding  of  the  NTDS  symbol  set.  The  set  was  also  expanded  to 
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include  symbols  for  “assumed  friend”  and  neutral  tracks.  A  North  Atlantic  Treaty  Organization 
(NATO)  committee  evolved  a  new  set  of  symbols  from  the  NTDS  set,  but  these  were  never 
formally  ratified. 

In  summarising  the  data  from  prior  studies,  the  paper  notes  that  some  of  the  findings  showing 
superiority  for  the  NATO  over  the  NTDS  set  were  only  obtained  when  the  two  symbol  sets  were 
presented  against  different  backgrounds,  and  that  in  general  no  conclusive  evidence  could  be  drawn 
over  which  set  was  better.  Therefore,  the  motivation  for  the  experimental  work  was  to  provide  a 
clearer  understanding  of  the  relative  strengths  and  weaknesses  of  the  NTDS  and  2525  symbol  sets 
in  order  to  recommend  improvements  to  the  design  concepts  laid  out  in  MIL-STD-2525B. 

The  methodology  required  subjects  (non  operators)  to  search  a  simulation  of  a  naval  tactical 
display  on  a  coloured  CRT  display  and  locate  specific  target  symbols  in  a  context  of  other  symbols 
and  against  a  plain  grey  or  more  complex  map  background.  Essentially  this  required  subjects  to 
count  the  number  of  a  specified  symbol  type  present  in  the  display.  Primary  measures  were  the 
time  required  to  locate  targets  and  performance  accuracy.  Four  experiments  were  reported;  each 
addressed  significant  issues  for  the  design  of  symbology. 

The  first  experiment  examined  the  ability  to  locate  prescribed  symbols  in  a  complex  background  as 
a  function  of  the  target  set  type  (NTDS  v  2525)  and  the  confusability  among  symbols.  The 
comparison  set  of  symbols  was  based  upon  the  nine  basic  NTDS  symbol  types.  The  results  showed 
that  performance  was  typically  better  with  a  colour  coded  target  set  and  that  the  NTDS  symbols 
produced  better  performance  than  the  2525.  However,  there  were  distinctive  patterns  of  results  for 
different  contact  symbols,  for  which  the  reader  should  consult  the  original  report. 

Experiment  two  examined  the  effect  of  background  type  and  symbol  complexity  (by  adding  icons 
which  indicated  platform  type)  on  performance  with  the  2525  symbol  set.  The  results  showed  that 
adding  an  icon  to  a  symbol  had  no  effect  on  performance  for  air  symbols,  but  degraded 
performance  for  the  surface  and  underwater  symbols.  Without  the  icons,  the  surface  symbols 
tended  to  be  more  detectable  than  the  air  and  underwater  symbols.  When  the  icons  were  added  this 
was  no  longer  the  case.  Further,  performance  on  these  two  categories  of  symbols  was  also  degraded 
by  the  addition  of  background  information. 

The  third  experiment  examined  the  ability  to  count  the  number  of  occurrences  of  a  specific  icon 
representing  a  “warfighting  object”  located  inside  one  of  the  three  hostile  symbol  shapes  (air, 
surface,  sub-surface)  in  a  cluttered  display.  In  general,  it  took  longer  to  locate  a  specific  icon  in  a 
symbol  shape  than  to  find  the  same  shape  without  an  internal  icon. 

The  fourth  experiment  examined  the  ability  to  identify  the  platform  associated  with  the  icon  inside 
the  shape  in  the  context  of  the  basic  search  task.  All  subjects  were  initially  trained  to  identify  the 
icons  to  a  level  of  95%  accuracy.  Overall,  there  was  no  significant  difference  in  performance  across 
the  individual  icon  types,  although  some  took  longer  to  identify  (air  symbols  with  rotary  and  fixed 
wing  icons  and  subsurface  symbols  with  the  sub,  mine,  and  decoy)  than  others. 

The  following  conclusions  can  be  drawn  from  these  experiments: 

•  Overall  performance  was  very  similar  on  the  colour  coded  NTDS  and  2525  symbols  sets. 
Thus,  when  the  symbols  were  presented  on  the  same  background,  and  the  recommended 
form  for  the  NTDS  symbols  was  used,  the  2525  symbols  showed  no  advantage  over  NTDS 
symbols.  The  use  of  the  same  background  and  thicker  lines  probably  contributed  to  the 
visibility  of  the  NTDS  symbols. 
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•  Both  symbols  sets  have  problems  that  were  not  identified  in  previous  comparisons  of  the 
NTDS  and  NATO  symbols.  When  all  nine  symbols  are  presented  simultaneously, 
performance  with  the  air  and  underwater  symbols  is  considerably  poorer  than  with  the 
surface  symbols. 

•  The  use  of  colour  coding  improves  the  detectability  of  the  symbols.  A  comparison  of  the 
monochrome  and  colour  coded  solutions  indicates  that  colour  coding  reduces  confusion 
across  the  different  shapes.  This  is  especially  true  for  the  2525  symbols. 

•  As  the  number  of  icons  increases  they  become  progressively  less  discriminable.  If  icons 
are  to  be  used  they  should  be  restricted  to  a  few  classes  only  and  reserved  for  special  cases 
(e.g.  critical  platform  types). 

•  The  visibility  of  the  2525  symbols  relative  to  the  NTDS  symbols  is  also  affected  by  the 
fact  that  all  the  2525  symbols  are  effectively  the  same  size.  The  larger  size  of  the  NTDS 
surface  symbols  contributes  substantially  to  their  visibility  and  reduces  the  likelihood  that 
they  will  be  confused  with  an  air  or  underwater  symbol  -  thus  contributing  to  the  visibility 
of  those  symbols.  Unfortunately,  reducing  the  size  of  some  of  the  2525  symbols  would 
make  them  less  suitable  for  adding  icons. 

The  report  makes  several  recommendations  on  how  to  increase  the  visibility  of  air  and  underwater 
symbols  and  provides  direction  for  further  studies  that  will  be  required  before  definitive 
recommendations  can  be  made  concerning  improvements  to  existing  recommendations  and 
guidelines  for  symbol  design. 

Discussion 

This  study  provides  an  important  review  and  critique  of  the  considerations  that  shape  the  design  of 
symbols  for  naval  tactical  displays.  The  historical  review  and  analysis  of  previous  work  provides  a 
clearer  understanding  of  the  strength  and  weaknesses  of  current  recommendations. 

The  experiments  conducted  provide  valuable  empirical  data  which  show  that  adoption  of  emerging 
ideas  on  symbol  design  and  the  implementation  of  iconic  information  into  symbols  are  complex 
matters  that  can  lead  to  design  solutions  which  result  in  performance  decrements,  as  well  as 
increments. 

The  results  clearly  point  out  the  need  for  a  significant  level  of  effort  to  be  expended  for  future 
empirical  investigations  of  symbol  design  alternatives  before  valid  and  robust  recommendations 
can  be  confidently  made  to  designers  of  symbol  sets  to  be  used  in  new  generations  of  naval  tactical 
displays.  The  results  also  point  to  the  need  for  a  greater  understanding  of  the  information  priorities 
of  operators  in  order  to  prescribe  important  candidate  contact  types  for  which  enhanced  contact 
information  should  be  provided  to  complement  the  basic  symbology. 

Limitations 

There  are  a  number  of  critical  variables  that  are  likely  to  influence  the  performance  with  tactical 
symbology  that  were  beyond  the  scope  of  these  studies.  Factors  such  as  ambient  lighting  (and 
range  of  lighting),  off-axis  viewing,  different  display  technologies,  backgrounds  and  overlays, 
scalability  under  different  range  settings  are  all  important  issues.  They  will  need  to  be  explored  in  a 
parametric  and  systematic  manner  in  order  to  produce  comprehensive  guidelines  that  can  be 
unambiguously  interpreted  by  future  system  designers. 
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7  Development  of  Concepts  of  Operation  for 
the  integration  of  automation  for  decision 
support 

7.1  Goals 

The  overarching  goal  of  this  work  was  to  generate  knowledge  on  how  MSDF  functionality  could 
be  practically  implemented  for  future  system  upgrades  to  the  Halifax  Class  C2  system.  The  sub¬ 
goals  were  as  follows: 

•  to  identify  the  types  of  decision  support  that  could  be  provided  by  MSDF  functionality  to 
support  selected  tasks  in  tactical  picture  compilation  and  track  management, 

•  to  provide  sample  CONOPS  for  the  proposed  support  functionality,  and 

•  to  provide  illustrative  examples  of  the  OMI  support  functionality. 

7.2  Responsible  parties 

This  work  was  contracted  to  Humansystems ®  Incorporated. 

7.3  Reports  produced 

Keeble,  A.  R.,  Matthews,  M.  L.,  Lamoureux,  T.M.  Berger,  N.  and  Zobarich,  R.  (2007).  Concept  of 
Operations  (CONOPS)  for  Recognition,  Standard  Identification  and  Track  Management  and 
Recommendations  for  Interface  Design  for  an  MSDF  Enhanced  System.  (DRDC  Toronto  No.  CR- 
2007-056),  Humansystems R  Incorporated,  Guelph,  ON. 
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7.4  Summary  of  reports 


7.4.1  Concept  of  Operations  (CONOPS)  for  recognition,  standard  identification 
and  track  management  and  recommendations  for  interface  design  for  an 
MSDF  enhanced  system 


Abstract 

The  goal  of  this  work  was  to  develop  a  Concept  of  Operations  (CONOPS)  and  prototype 
Operator  Machine  Interface  (OMI)  for  the  provision  of  a  semi-automated  and  automated 
decision  support  capability  based  upon  multi-source  data  fusion  (MSDF)  and  evidential 
reasoning  algorithms  to  assist  the  processes  of  contact  detection,  recognition  and  identification 
performed  by  the  Halifax  Class  frigate.  The  report  starts  by  documenting  the  existing,  baseline 
CONOPS  for  air  and  surface  picture  compilation  and  then  maps  onto  this  opportunities  where 
MSDF  could  play  a  role.  Based  upon  such  opportunities,  a  CONOPS  for  MSDF  enhanced 
decision  support  was  developed  and  an  experimental  OMI  was  prototyped.  The  CONOPS  and 
OMI  were  then  validated  with  Navy  Subject  Matter  Experts  (SMEs)  and  positive  support  to  the 
proposals  was  obtained.  The  proposed  OMI  and  CONOPS  provides  enhanced  support  to 
overall  contact  management,  improved  situation  awareness  for  contacts  of  interest,  collated 
evidence  and  recommendations  for  recognition  and  alerts  for  potential  recognition  errors.  An 
interactive  PowerPoint  demonstration  of  the  proposed  OMI  was  delivered. 


Summary 

The  initial  phase  of  this  project  involved  an  analysis  of  the  existing  CONOPS  for  the  conduct  of 
recognition,  standard  identification  and  track  management  for  air  and  surface  contacts.  The 
analysis  was  based  upon  information  collated  from  interviews  with  Navy  SMEs,  the  experience  of 
one  Navy  SME  attached  to  the  project  (former  sea  trainer  and  ORO)  and  reports  of  HF  analyses 
conducted  under  the  COMDAT  TD  program.  The  following  major  functions  that  could  be 
impacted  by  MSDF  technology  were  identified: 

•  equipment  set-up; 

•  radar  track  initiation; 

•  contact  localisation  and  tracking  (single  and  multiple  ships); 

•  track  correlation; 

•  association  of  EW  to  vehicular  tracks  (single  and  multi-ship); 

•  the  recognition  process:  considerations  of  speed,  ESM,  Identification  Friend  or  Foe  (IFF), 
visual  information; 

•  the  recognition  process:  considerations  of  LINK  11,  Primary  ID  (PRI  ID),  ID  amplification 
(ID  AMP),  Global  Command  and  Control  System  (GCCS)-  platform  and  Electronic 
Intelligence  (ELINT)  tracks; 

•  recognition  confidence  levels  and  decision  making; 

•  the  Standard  Identification  process:  definition  and  assignment; 
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•  the  Standard  Identification  process:  ID  amplification,  LINK  11,  PRI  ID  and  ID  AMP; 

•  track  numbering;  and 

•  team  situation  awareness. 

The  second  step  involved  a  consideration  of  how  MSDF  functionality  could  support  the  above 
operations.  For  the  purpose  of  the  analysis,  this  consideration  was  not  constrained  by  the  actual 
functionality  in  the  TD  prototype  but  was  based  on  a  reasonable  extension  to  what  MSDF  could  be 
capable  of  within  the  bounds  of  known  technology.  The  scoping  of  this  functionality  involved 
discussion  with  and  feedback  from  the  Scientific  Authority. 

As  a  result  of  this  analysis,  a  number  of  MSDF  support  functions  were  identified  and  the  operator 
information  needs  to  make  use  of  such  functions  were  prescribed.  This  information  was  then  fed 
into  design  seeds  for  system  functions  and  a  CONOPS  to  support  MSDF. 

The  philosophy  was  adopted  to  use  decision  support  conservatively  in  order  to  maximise  operator 
trust  and  acceptance  and  to  provide  an  evolutionary  change  in  the  C2  CONOPS  (as  opposed  to  an 
aggressive  application  that  took  operators  out  of  the  loop  and  gave  decision  making  entirely  to 
MSDF).  The  recommendations  for  a  conservative  integration  of  MSDF  functionality  assumed  no 
change  in  the  existing  command  authority  to  make  decisions  and  recommendations.  The  proposed 
CONOPS  was  designed  to  function  within  a  multi-ship  formation  where  not  all  ships  would 
necessarily  be  equipped  with  COMDAT  decision  support.  As  an  evolutionary  system,  the  existing 
authority  structures  for  assessments  of  Standard  ID  and  recognition  were  retained.  Any  automation 
regarding  ID  or  recognition  was  limited  and  specifically  selected  by  the  operators  in  their  particular 
context. 

The  following  assumptions  were  made  concerning  the  MSDF  functionality: 

•  a  database  of  all  air  and  surface  vessels, 

•  MSDF  data  fusion  algorithms  that  accurately  combine  sensor  information, 

•  recognition  propositions  based  upon  combined  sensor  information  and  evaluated  by 
evidential  reasoning  algorithms, 

•  an  accurate  and  independent  database  of  all  ongoing  information  amassed  within  the 
MSDF  system,  and 

•  a  capability  for  the  MSDF  data  fusion  algorithms  to  accept  operator  input  and  use  the  data 
in  a  similar  manner  to  sensor  sources. 

The  following  principles  were  adopted  as  a  high  level  CONOPS: 

•  Operators  will  typically  use  the  MSDF  functionality  to  consolidate  evidence  for 
assessments  which  they  will  continue  to  make  themselves  (apart  from  those  instances 
where  the  MSDF  decision  aid  takes  ID  or  recognition  action  automatically). 

•  Depending  on  the  authority,  or  the  operator  in  question  and  the  authority  structure  in  force 
for  the  operation,  the  operator  may  just  review  the  evidence  provided  by  the  aid  and  take 
action  independently  to  accept  the  propositions  and  send  them  to  the  CCS  Global 
Database  (GDB). 


Humanly  .stem  .s’  Incorporated 


Final  COMDAT  Human  Factors  Report 


Page  59 


•  Operators  may  use  the  evidence  to  make  an  assessment  report  up  the  authority  chain, 
sending  proposition  reports  through  to  the  CCS  as  appropriate  after  feedback  from  the 
higher  authority. 

Several  assumptions  were  made  concerning  the  relationship  between  the  MSDF  and  CCS  systems: 

•  MSDF  will  be  integral  to  the  CCS,  not  a  stand-alone  system. 

•  Certain  data  fields  in  the  CCS  GDB  will  be  fed  real  time  from  the  MSDF  algorithms 
(course  and  speed  of  a  contact  being  tracked  locally  by  a  MSDF  ship  as  determined 
directly  by  the  MSDF  tracking  algorithms). 

•  Others  data  fields  will  not  have  this  direct  link.  (ID,  PRI  AMP  and  ID  AMP). 

Central  to  the  proposed  CONOPS  is  the  notion  of  Operator- As-Input  to  provide  data  to  the  MSDF 
algorithms  that  are  held  by  the  team  but  not  available  to  MSDF  through  other  means.  The 
CONOPS  relies  heavily  on  the  ability  of  operators  to  interact  with  recognition  propositions 
presented  by  the  system,  or  even  with  recognition  possibilities  not  presented  by  the  system  but 
resident  in  the  MSDF  database,  to  enhance  the  relevance  and  usefulness  of  the  system.  This 
interaction  would  be  purposeful  and  undertaken  within  an  authority  structure  such  that  the 
interaction  should  almost  always  result  in  an  improvement  in  the  data  and  thus  the  output  from  the 
aid.  The  functionality  for  an  operator  to  review  the  results  of  an  interaction  and  then  “undo”  it 
allows  them  to  troubleshoot  or  “what  if’  as  they  work  through  the  recognition  challenge. 

An  important  element  of  this  interaction  is  the  continued  parallel  processing  of  all  data  being  fed 
into  the  MSDF  system  without  operator  interaction,  which  facilitates  the  identification  and 
presentation  of  disconfirming  information  to  the  assessments  being  made  by  the  operators. 
Allowing  MSDF  to  continue  processing  its  organic  data  (all  data  except  the  operator  initiated  data) 
after  an  operator  has  interacted  with  the  propositions  allows  for  disconfirming  evidence  to  be 
presented,  so  that  if  the  aid  is  clearly  assessing  an  ID  or  recognition  differently  from  the  one  in  the 
Operator- As-Input  process,  an  operator  can  review  the  evidence. 

The  following  operational  functions  were  identified  as  being  the  most  likely  to  benefit  by  the 
proposed  MSDF  decision  support  capability: 

•  improved  situation  awareness  of  the  tactical  picture, 

•  improved  situation  awareness  for  priority  contacts, 

•  integration  of  all  relevant  data  on  a  contact, 

•  enhanced  capability  under  overload  conditions, 

•  improved  association  of  EW  to  platforms,  and 

•  validation  of  recognition  assessments. 

On  the  basis  of  this  overall  philosophy  concerning  the  application  of  decision  support,  the  proposed 
CONOPS  and  the  identification  of  operator  information  requirements,  a  demonstration  OMI  was 
developed  based  on  the  design  seeds.  The  design  team  consulted  guidelines  developed  for  C2  OMI 
as  part  of  the  COMDAT  TD  program,  reviewed  recent  literature  on  new  developments  in  design 
concepts  and  consulted  other  relevant  HF  guidelines  in  shaping  the  proposed  OMI  design. 
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The  report  then  provides  details  of  the  major  functionality  and  OMI  design  details  by  presenting 
figures  and  a  narrative  text  to  describe  what  the  functionality  does  and  how  it  is  to  be  used  by 
operators.  Essentially,  the  functionality  comprises  four  components: 

1 .  Setup  page  for  configuring  the  system. 

2.  Interactive  data  pages  through  which  the  operator  compiles  the  picture  and  conducts 
recognition  assessments. 

3.  Alerts:  these  are  operator  configured  to  provide  improved  situation  awareness  under 
prescribed  conditions. 

4.  A  track  symbol  and  data  block  (tactical  display). 

The  proposed  functionality  went  through  two  validation  processes  with  Navy  SMEs.  An  initial 
validation  after  the  design  philosophy  had  been  developed,  and  a  final,  more  in  depth  validation,  in 
which  the  actual  OMI  and  functions  were  systematically  reviewed  and  analysed  by  a  representative 
cross  section  of  Ops  Room  operators.  This  final  validation  of  the  proposed  functionality  received 
moderate  to  strong  endorsement  from  the  SMEs  and  minor  suggestions  for  improvement  were 
identified  and  incorporated  into  a  subsequent  design  revision. 

In  addition  to  the  report,  an  interactive  PowerPoint  demonstration  was  delivered  which  provides  a 
tutorial-like  package  that  illustrates  how  the  functionality  would  work  and  how  operators  would 
use  it. 

Discussion 

This  work  provides  an  important  step  in  translating  how  an  MSDF  and  evidential  reasoning  support 
system  might  be  integrated  in  a  very  practical  way  into  a  future  re-design  of  the  Halifax  Class  C2 
system.  The  proposed  evolutionary  approach  to  the  system  design  makes  good  sense  and  would 
allow  for  an  incremental  assignment  of  decision  making  to  automated  functions  if  operational 
experience  were  to  warrant  it. 

The  proposed  design  makes  some  interesting  recommendations  for  system  alerts  and  improved 
situation  awareness  (which  are  not  necessarily  tied  to  an  MSDF  capability),  which  may  merit 
further  consideration  as  HMCCS  evolves. 

The  analyses  of  the  existing  CONOPS  in  this  report  would  be  of  use  for  future  system  (re)designers 
in  understanding  how  the  Ops  Room  team  currently  conducts  the  processes  of  surface  and  air 
contact  recognition  and  standard  identification. 

One  significant  limitation  is  that  the  underlying  MSDF  functionality  to  support  the  proposed 
CONOPS  does  not  currently  exist  within  the  COMDAT  TD. 
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8  Support  to  DRDC  Atlantic  for  land  and  sea 
trials 


8.1  Goals 

The  goals  of  this  work  were  to 

•  Provide  a  trial  plan  to  enable  the  conduct  of  real-time,  HIL  trials  to  evaluate  the 
performance  of  the  TD. 

•  Identify  scenarios  that  would  be  used  in  HIL  trials  that  would  be  suitable  for  exercising  the 
MSDF  TD  capability. 

•  Support  the  conduct  of  a  HIL  trial  to  assess  MSDF  functionality. 

The  third  goal  was  never  met  because  of  a  change  in  emphasis  of  the  TDP. 


8.2  Responsible  parties 

This  work  was  contracted  to  Humansvste/ws "  Incorporated. 


8.3  Reports  produced 

Matthews,  M.  L.  and  Keeble,  A.  R.  (2004).  Towards  a  trial  plan  for  evaluating  the  COMDAT  TD: 
Final  report.  (DRDC  Toronto  No.  CR2004-053).  1 1  \xmansystems  Incorporated,  Guelph, 
ON. 

8.4  Summary  of  reports 

8.4.1  Towards  a  trial  plan  for  evaluating  the  COMDAT  TD:  Final  report 


Abstract 

This  report  outlines  a  series  of  evaluations  and  analyses  of  the  COMDAT  TD  with  a  view  to 
conducting  future  trials  to  evaluate  its  potential  impact  upon  operator  performance  in  the 
Operations  Room  of  the  Halifax  Class  frigate.  Specific  issues  commented  upon  include  the 
operator-machine  interface,  the  logistics  of  integrating  the  TD  into  a  suitable  trial 
environment,  the  availability  of  existing  scenario  elements  to  provide  a  suitable  evaluation 
context,  the  types  of  performance  measures  that  could  be  feasibly  implemented  and  options 
for  the  format  and  location  of  future  trials. 


Summary 

The  report  covers  the  following  major  areas: 
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•  an  outline  of  aspects  of  system  usability,  functionality  and  operational  performance  to  be 
assessed  in  planned  sea  and  land-based  trials; 

•  comments  upon  the  existing  TD  functionality  and  its  implications  for  an  evaluation  trial; 

•  assessment  of  the  availability  of  existing  scenarios  to  be  used  in  future  trials  to  evaluate  the 
TD; 

•  provision  of  an  overall  trial  plan  for  evaluating  the  TD; 

•  identification  of  requirements  for  data  capture  and  OMI  enhancements  in  order  to  conduct 
a  future  evaluation  trial;  and 

•  recommendations  for  the  next  steps  to  be  taken  in  the  overall  trial  plan. 

The  trial  plan  proposed  an  incremental  approach  to  T&E  with  each  step  designed  to  evaluate  some 
aspect  of  the  TD.  Step  1  would  involve  an  initial  proof  of  capability  assessment  that  would  involve 
a  systematic  test  of  the  TD  functionality  in  which  simple,  but  representative,  operational  events  are 
used  to  stimulate  the  TD  algorithms,  using  mini  scenarios  that  can  be  readily  and  quickly 
constructed.  It  would  also  establish  the  limits  and  capabilities  of  the  TD  and  define  an  operating 
envelop  within  which  to  assess  the  specific  functionality.  The  second  step  would  involve  SME 
evaluation  of  the  functional  utility  of  the  system  and  development  of  a  CONOPS.  The  process 
recommended  would  be  a  scenario,  or  event-based,  cognitive  walkthrough.  The  third  step  would  be 
full  HIL  trials  in  which  system  functionality  would  be  assessed  with  complete  teams  working 
through  a  predetermined  scenario  in  real  time  with  concurrent  capture  of  MOP  data.  For  each 
component  of  the  trial  plan,  the  report  provides  details  on  logistical  requirements,  the  MOPs  to  be 
collected,  and  the  methods  for  collecting  MOP  data. 

The  report  identifies  several  existing  scenarios  that  are  used  for  ORTT  training  that  would  provide 
a  suitable  context  for  conducting  HIL  trials.  These  scenarios  provide  the  required  event 
characteristics  to  assess  a  wide  range  of  the  TD  functionality  and  its  impact  on  operational 
performance.  It  also  identifies  a  number  of  logistical  problems  that  would  be  faced  in  integrating 
the  TD  into  potential  test  environments,  such  as  the  ORTT.  These  include  issues  of  OMI 
consistency,  integration  of  MSDF  tracks  with  CCS  tracks  as  well  as  technical  issues  concerning 
interactions  among  the  underlying  software  systems. 

Discussion 

The  proposed  plan  represents  a  standard  approach  to  conducting  a  HF  evaluation  of  a  prototype 
system.  The  use  of  an  incremental  approach  allows  design  flaws  to  be  identified  early  in  the 
evaluation  process  prior  to  substantive  investment  in  resources.  Since  the  evaluation  was  never 
carried  out,  it  is  not  possible  to  say  if  this  approach  works  for  systems  such  as  the  COMDAT  TD. 
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This  section  provides  an  overall  appreciation  of  the  relevance  of  the  HF  work  conducted  under 
COMDAT,  lessons  learned,  and  areas  for  future  research.  As  stated  in  the  introduction,  a  great  deal 
of  the  HF  work  under  COMDAT  was  not  necessarily  directly  related  to  the  development  and 
evaluation  of  the  TD.  Its  goal  was  to  increase  DRDC  and  industry  capability  to  provide  HF  support 
to  R&D  projects  such  as  COMDAT  through  the  development  of  improved  tools  and  methods. 
Under  the  TDP,  methods  for  conducting  requirements  analysis,  measuring  human  performance  in 
complex  systems,  and  providing  design  guidance  were  explored.  As  a  result,  we  now  have  a  better 
understanding  of  the  utility  of  the  methods  that  were  investigated  as  well  as  considerably  more 
knowledge  about  the  information  requirements  and  processes  involved  in  building  the  tactical 
picture  in  a  naval  C2  system.  In  addition,  we  have  generated  improved  guidance  for  designing  the 
interfaces  needed  to  present  the  required  information  and  support  these  processes.  As  is  usually  the 
case  with  R&D,  we  also  have  a  better  understanding  of  what  research  still  needs  to  be  done  to 
allow  us  to  build  more  effective  interfaces  and  decision  support  systems  for  naval  C2. 

9.1  Requirements  analyses 

9.1.1  Utility  of  the  different  methods 

As  shown  in  Section  3,  each  of  the  analyses  provided  useful  and  in  many  cases  similar  information 
about  the  goals,  tasks,  information  requirements  and  processes  used  by  operators  in  the  Halifax 
Class  Ops  Room.  Although  the  analyses  varied  in  their  coverage,  the  overlap  does  provide  unique 
insight  into  the  utility  of  the  different  methods  for  this  type  of  application.  Each  method  has 
strengths  and  weaknesses  that  might  make  it  the  method  of  choice  under  specific  conditions,  but 
any  of  the  methods,  if  used  properly  by  trained  personnel,  would  probably  be  suitable.  However,  it 
is  clear  that  both  cognitive  and  task  requirements  should  be  included  and  that  it  is  important  to 
identify  the  higher  level  system  goals  as  well  as  the  specific  goals  of  the  individual  operators  and  to 
understand  the  communication  links  amongst  the  various  members. 

9.1.2  Representation  of  information  from  analyses 

There  is  a  continuing  challenge  of  how  to  make  the  extensive  results  of  the  HF  analyses  conducted 
under  the  TD  amenable  to  the  needs  of  those  responsible  for  system  re-design  and  requirements 
specification.  While  this  report  provides  an  overall  summary  of  the  analyses,  it  is  structured  in 
terms  of  what  was  done  under  the  different  contracts  rather  than  providing  a  user-oriented 
document.  The  latter  would  focus  more  on  questions  that  a  user  of  the  information  might  be 
expected  to  ask.  While  the  answer  to  such  questions  will  certainly  be  found  in  the  report,  there 
needs  to  be  a  mapping  to  allow  more  direct  access  to  the  information  based  upon  a  query-oriented 
architecture. 

9.1.3  Tools 

Although  the  methods  for  conducting  a  requirements  analysis  are  reasonably  well  defined,  the 
conduct  of  these  analyses  still  tend  to  be  extremely  labour  intensive.  There  is  a  need  for  better  tools 
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to  support  data  collection,  data  representation,  and  the  transformation  of  the  information  from  data 
collection  tools  such  as  Task  Architect  to  simulation  tools  such  as  IPME. 

9.1.4  Further  analysis  work 

Although  the  analyses  were  extensive,  there  were  several  areas  that  were  not  fully  investigated  by 
any  of  the  analyses  and  should  be  pursued  further.  Most  of  these  have  been  noted  in  the  specific 
reports  and/or  Section  3.  They  include 

•  Communications:  While  several  analyses  noted  the  overload  on  the  auditory  channel, 
bottlenecks  at  several  operator  positions,  and  communication  errors,  no  systematic  analysis 
has  been  conducted  at  a  sufficient  level  of  detail  to  understand  the  extent  and  nature  of  the 
communication  problems.  There  is  a  need  to  conduct  both  a  lateral  and  downward  flow 
analysis  to  complement  the  upward  flow  analysis  in  the  HGA.  The  most  direct  and  valid 
method  for  analysing  the  communication  process  would  be  to  examine  the  communication 
records  obtained  during  ORTT  training.  These  records  contain  a  very  detailed  log  of  all 
communications  by  all  team  members  for  all  stages  of  picture  compilation  and  threat 
response  which  would  provide  a  quantitative  basis  for  specifying  the  type  and  frequency  of 
communication  bottlenecks  and  communication  errors  by  task  and  operator  position. 

•  Goal  analysis:  There  needs  to  be  a  greater  understanding  of  goal  conflict  particularly  as  it 
relates  to  management  of  the  picture  compilation  process.  The  information  requirements  to 
provide  decision  support  to  help  resolve  goal  conflict  need  to  be  identified.  There  is  also  a 
need  to  decompose  the  goal  analysis  to  a  finer  grain  to  allow  a  better  understanding  of  the 
goals  associated  with  the  compilation  of  the  tactical  picture. 

9.2  Measuring  human  performance  in  complex  systems 

As  the  cost  and  complexity  of  new  systems  increase  and  examples  of  unacceptable  systems 
proliferate,  there  is  a  growing  demand  that  developers  prove  that  their  systems  will  actually 
improve  overall  performance.  Thus,  a  significant  portion  of  the  HF  work  under  COMDAT  was 
directed  at  improving  our  capability  to  conduct  HIL  studies.  The  results  of  this  work  are 
summarized  in  Sections  4,  5,  and  8.  Prior  to  COMDAT,  there  had  been  a  substantive  effort  to 
develop  a  process  for  conducting  HF  evaluations  of  C2  systems  (Matthews  et  al.  1997).  COMDAT 
was  one  of  the  first  opportunities  to  apply  this  process.  The  experience  indicated  that  lack  of  human 
resources  and  suitable  facilities  make  it  unlikely  that,  in  the  near  term,  we  will  to  able  to  conduct 
the  type  and  quality  of  evaluation  that  HF  specialist  would  traditionally  favour  in  order  to  collect 
robust,  reliable,  and  valid  data.  However,  we  have  improved  our  ability  to  define  measures  of 
performance  and  validate  them.  We  also  have  a  better  understanding  of  what  are  the  barriers  and 
potential  alternatives  to  traditional  HF  research  paradigms. 

9.2.1  Measures  of  performance 

The  most  successful  part  of  this  project  was  the  development  and  validation  of  the  MOPS 
(Matthews  et  al.  2004a;  b).  As  a  result,  developers  have  a  clear  process  for  how  to  develop  MOPs 
and  a  proven  method  for  validating  them.  The  first  step  is  to  ensure  that  MOPS  and  a  potential 
method  of  assessing  each  are  developed  as  part  of  the  requirements  analysis.  Beyond  the  time  and 
effort  saved  in  having  to  go  back  to  SMEs  at  a  later  stage,  this  process  ensures  that  the  MOPs  and 
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evaluation  methods  are  available  early  in  the  development  process  prior  to  building  a 
prototype/TD.  Hopefully  this  will  lead  to  the  necessary  measurement  capability  being  an  integral 
part  of  the  TD  and  not  an  afterthought. 

Since  the  MOPs  reflect  current  goals  and  tasks,  it  should  be  feasible  to  validate  them  using  the 
process  followed  in  the  studies  reported  in  Section  5  (Matthews,  Keeble,  Bruyn  Martin,  and  Sartori 
2007b).  This  approach  provides  information  on  which  MOPs  are  likely  to  be  measurable  and 
relevant  data  on  performance  on  the  current  system.  Moreover,  the  results  show  the  “actual” 
practice  in  operation  as  opposed  to  the  “theoretical”  process  identified  in  the  requirements  analyses 
(which  are  based  on  SME  accounts  and  analyses  of  documentation).  Thus,  the  operational  data 
often  reveal  short  cuts  and  the  potential  for  errors  in  the  process.  This  information  can  often  lead  to 
insight  for  the  design  of  new  technology. 

9.2.2  Test  and  evaluation-  lessons  taught 

One  key  objective  of  the  TD  was  to  conduct  an  empirical  evaluation  of  the  effectiveness  of  the 
MSDF  functionality  in  the  context  of  operational  tasks  and  user  requirements  (Matthews  and 
Keeble  2004).  To  this  end,  appropriate  MOPS  were  developed  with  the  ultimate  goal  of  utilising 
these  in  a  full  scale  evaluation  in  a  simulated  operational  context.  The  evaluation  trial  was 
conceived  of  as  being  a  close  approximation  to  the  team  and  individual  tasks  involved  in  target 
detection  and  recognition,  ft  would  be  conducted  in  a  test  environment  that  was  of  sufficient 
operational  fidelity  that  operators  could  access  the  TD  functionality  in  a  manner  that  was  integrated 
into  their  standard  operating  environment. 

The  desired  objective  of  conducting  a  full  scale  evaluation  with  operators  produced  many 
challenges  at  the  technical,  engineering,  logistic,  and  personnel  levels.  For  example,  at  the 
engineering  level  there  were  considerations  of  how  to  display  MSDF  tracks  and  integrate  them  into 
the  existing  CCS  tactical  picture  while  maintaining  track  numbering  integrity.  There  were  also 
issues  of  the  choice  of  symbology  and  the  representation  of  certainty  information.  There  was  a 
further  difficulty  in  being  able  integrate  the  TDP  functionality  in  a  transparent  manner  into  the 
aging  infrastructure  of  the  existing  CCS.  Another  issue  was  that  the  TDP,  which  was  designed  to 
implement  and  demonstrate  a  particular  technological  concept,  did  not  have  the  features  required  to 
meet  the  requirements  of  HIT  T&E;  it  lacked  the  necessary  data  capture  capability  and  ability  to 
flexibly  change  system  parameters  to  allow  manipulation  of  variables  of  interest  for  T&E  purposes. 
At  the  logistical  level,  there  were  difficulties  in  finding  a  suitable  T&E  environment,  configuring 
that  environment,  creating  scenarios  and  finding  available,  experienced  Ops  Room  personnel  to 
participate  in  a  trial,  or  series  of  trials.  At  the  personnel  level,  there  were  issues  concerning  the 
need  for  a  CONOPS  for  the  new  functionality,  the  training  in  such  a  CONOPS  and  how  this  would 
mesh  with  existing  operational  procedures  and  practices. 

Given  such  challenges  to  conducting  evaluations  of  fully  evolved  systems  in  operational  or  quasi- 
operational  contexts  (which  are  likely  to  be  found  in  other  situations  where  new  military  systems 
are  being  developed),  alternate,  and  more  practical  and  cost-effective  approaches  will  need  to  be 
considered.  In  this  respect,  the  approaches  suggested  by  Matthews  (1997),  Osga  (1995),  and 
Kellmeyer  and  Osga  (2000)  may  have  some  merit. 

Based  on  the  approach  of  Matthews  et  al.  (1997)  and  integrating  the  lessons  learned  from  more 
recent  experiences  in  this  and  other  projects,  it  is  recommended  that  the  general  approach  to  T&E 
should  focus  on  the  following  aspects  of  a  C2  system: 
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•  Utility  (the  usefulness  of  the  system  functions  for  critical  operational  tasks;  support  for 
communication,  situation  awareness,  situation  analysis,  decision  making). 

•  User  fit  (meets  operator  requirements  for  usability,  OM1  design,  configurable  to  different 
operator  roles,  positions  and  teams,  training  and  suitable  workload). 

•  Operator-system  performance  (on  critical  tasks). 

•  Operational  capability  (team  based  critical  tasks  in  an  operational  context;  workload;  trust 
in  automation). 

•  Overall  system  capability  (performance  envelope;  resource  requirements). 

With  these  evaluation  dimensions  in  mind,  the  approach  advocated  is  to  conduct  much  of  the  basic 
evaluation  for  utility  and  user  fit  (particularly  usability)  at  a  very  early  stage  in  development  using 
storyboards  and  very  simple  mockups.  This  approach  is  a  cost  effective  way  of  getting  early 
feedback  on  whether  basic  operator  requirements  are  being  met,  and  avoids  too  much  effort  being 
placed  on  developing  a  fully  interactive  prototype  to  essentially  obtain  the  same  outcome 
(Kellmeyer  and  Osga  2000).  In  the  case  of  the  TD,  storyboards  were  used  for  evaluating  concepts 
for  representing  uncertainty  (Unger  Campbell  and  Baker  2003)  as  well  as  for  the  CONOPS  design 
concepts  to  support  MSDF  functionality  (Keeble  et  al.  2007). 

With  such  feedback  integrated  into  the  development  cycle,  the  next  stage  of  testing  would  be  to 
validate  the  ensuing  design  solutions  with  functional  prototypes  and  to  conduct  operator-system 
performance  testing  on  critical  tasks  using  appropriate  performance  metrics  (both  of  which  have 
been  identified  by  prior  HF  analyses).  The  approach  to  this  phase  of  testing  would  be  to  use  the 
simplest  and  most  basic  interactive  functionality  to  gather  the  required  information.  This 
evaluation  would  eventually  be  extended  to  include  sequential  task  strings  that  replicate  core 
operational  procedures  by  individuals  and,  possibly,  teams. 

Using  such  an  approach  would  mean  that  cost  risk  is  mitigated  for  the  developer  and  the  match  to 
operational  requirements  for  performance  and  user  fit  is  mitigated  for  the  operational  community. 

It  would  be  anticipated  that  90-95%  of  the  HF  issues  concerning  the  new  system  could  be 
addressed  in  this  way,  leaving  a  small  number  of  issues  concerning  operational  fit  to  be  addressed 
once  a  deliverable,  functional  system  is  fielded  to  the  operational  environment.  Some  of  the 
critical  issues  that  may  only  be  validly  evaluated  in  an  operational  environment  include  trust  in 
automation,  workload,  and  task  management.  Essentially,  this  general  approach  can  be  though  of  as 
a  sophisticated  implementation  of  “build  a  little-test  a  little”. 

It  should  be  noted  that  even  with  such  an  approach,  it  may  still  not  be  possible  to  fully  evaluate  the 
overall  system  capability  or  bandwidth  because  of  logistical  or  other  constraints  in  the  typical 
testing  or  operational  environment.  For  example,  if  there  was  a  need  to  look  at  performance  when 
there  is  a  mass  attack  and  a  high  number  of  tracks  to  be  processed  (a  scenario  for  which  MSDF 
capability  has  been  advocated  as  providing  a  benefit),  this  could  not  be  done  with  existing  scenario 
capabilities  and  would  be  limited  by  the  existing  procedures  and  training  standards  of  personnel. 
Also,  if  there  was  a  desire  to  empirically  evaluate  the  effects  of  manpower  reductions  in  the  Ops 
Room,  or  a  change  in  responsibilities  or  procedures,  this  would  be  very  difficult  to  implement  in  a 
HIL  trial.  In  such  cases,  modelling  and  simulation  provide  an  additional  T&E  approach  that  could 
successfully  address  these  and  other  similar  issues.  Such  an  approach  should  be  seriously 
considered  in  the  development  of  complex  systems  in  which  there  is  a  desire  to  obtain  sound 
evidence  on  the  degree  to  which  the  system  will  improve  operational  performance  and  how  all  of 
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the  system  elements  (human,  hardware,  and  software)  can  be  fine  tuned  and  manipulated  to 
maximise  the  operational  gain. 


A  summary  of  the  sequential  stages  of  evaluation  are  outlined  in  the  following  table. 


T&E  requirements 

Approach 

Example 

Requirements;  system  analysis 

Prior  to  system  development 

Work,  function,  cognitive,  task  and 
Hierarchical  Goal  Analyses;  OMI 
Guidelines 

Utility,  usability;  OMI  design 

Storyboards,  mock-ups,  table  top 
evaluation,  cognitive  walkthroughs 

Symbology;  display  of  MSDF  data; 
integration  with  CCS  tracks;  alerts; 
representing  certainty 

Performance  evaluation-  core 
functionality-single  tasks;  dynamic 
usability 

Appropriate  MOPS  for  conducting 
simple  critical  tasks;  screen 
prototypes,  interactive,  dynamic 
displays 

Single  contact  recognition  and 
identification 

Operational  procedures:  individuals 
and  teams 

Interactive  functionality  to  support 
performance  of  task  strings; 

CONOPS 

Tactical  picture  compilation,  individual 
situational  awareness,  prioritisation  of 
targets 

Operational  capability 

Interactive  scenarios  using  team 
based  activities  for  tactical  picture 
compilation 

Validation  of  full  MSDF  functionality  in 
a  more  dynamic  and  realistic 
environment;  team  situation 
awareness;  time  required  for  contact 
recognition;  trust  in  MSDF  automation 

Overall  system  capability- 
performance  envelop 

Modeling  and  simulation 

Impact  of  MSDF  with  high  contact 
rates;  reduced  manning;  operator 
overload 

To  summarise,  the  logistical  and  other  requirements  to  support  a  full  scale  trial  dedicated  to  T&E 
were  shown  over  the  course  of  the  TD  to  be  almost  insurmountable  given  the  many  constraints 
identified.  However,  the  training  environment  of  the  ORTT  was  found  to  provide  an  opportunity  to 
collect  the  required  operationally  valid  data  (and  has  been  demonstrated  for  the  existing  processes 
of  air  and  surface  contact  detection,  recognition  and  identification)  (Matthews  et  al.  2007b).  For 
the  future,  an  ability  to  integrate  new  C2  functionality  into  the  ORTT  system  would  represent  an 
effective  strategy  for  evaluating  that  functionality  at  a  high  level  of  fidelity  and  with  complex  tasks. 

For  the  evaluation  of  the  utility  and  usability  of  new  system  functions,  a  more  practical  and  cost- 
effective  approach  is  recommended  using  cognitive  walkthroughs  of  early  design  concepts  and 
simple  prototypes  of  system  functions.  Subsequently,  this  would  be  followed  by  an  approach  that 
employed  dynamic  displays  with  limited  functionality  to  allow  operator  interaction  for  the  conduct 
of  core  critical  tasks. 

Overall,  the  goal  of  the  T&E  strategy  must  be  to  optimize  the  method  and  approach  to  evaluation 
by  using  just  enough  technology  to  address  the  issues  of  interest.  Fundamentally,  this  means  that 
there  is  a  need  to  develop  a  priori  a  detailed  T&E  plan  to  identify  these  issues  and  to  identify  the 
appropriate  approach  and  logistics  for  an  incremental  process  of  evaluation.  The  risks  of  relying 
solely  on  a  final  evaluation  of  a  completed  system  to  accomplish  the  various  needs  of  T&E  are 
high,  and  even  if  they  can  be  mitigated  through  a  careful  analysis  of  the  logistics  and  practicality  of 
such  an  approach,  represent  a  high  investment  in  effort  with  a  low  probability  of  successfully 
achieving  the  required  level  of  comprehensiveness,  validity  and  quality  in  the  evaluation  data 
obtained. 
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9.3  System  design/re-design 


9.3.1  Use  of  a  style  guide 

In  recent  years,  many  style  guides  have  been  developed  (Unger  Campbell  2001b).  They  are  seen  as 
a  way  of  avoiding  the  development  of  incompatible  interfaces  across  multiple  systems  in  complex 
environments  such  as  an  Ops  Room.  Although  it  proved  relatively  straightforward  to  develop  the 
COMDAT  OM1  Style  Guide  and  the  concept  was  readily  accepted  by  the  Navy,  it  was  never 
actually  used  in  the  development  of  the  interface  for  the  TD.  Instead,  the  designers  based  their 
interface  on  the  existing  CCS  with  all  its  known  flaws.  Thus,  it  is  not  possible  to  assess  the  actual 
utility  of  the  document. 

The  review  of  the  TD  interface  by  the  developer  of  the  style  guide  (Unger  Campbell  2004b) 
indicated  that  it  did  contain  most  of  the  necessary  guidance  needed  to  evaluate  the  interface. 
However,  the  developer  was  familiar  with  the  content  and  the  format.  Thus,  a  naive  user  might 
have  had  greater  difficulty.  Ideally,  the  guide  would  be  used  during  the  development  phase.  As 
indicated  in  the  discussion  in  Section  6,  the  guide  may  be  less  useful  for  that  purpose  because  it 
lacked  higher  level  guidance  and  sufficient  relevant  examples.  Possibly  even  more  critical  is  the 
lack  of  a  supporting  software  package  which  would  allow  the  designer  to  easily  generate  compliant 
windows,  menus,  etc.  This  type  of  tool  allows  the  designer  to  focus  on  the  overall  design  and 
automatically  ensures  a  more  consistent  and  compliant  look  and  feel. 

9.3.2  Interface  improvements 

The  COMDAT  reports  contain  a  large  number  of  recommendations  for  improving  the  current 
interface  and  for  designing  an  interface  for  a  CCS  with  MSDF.  Unfortunately,  none  of  these  were 
implemented.  However,  all  of  the  recommendations  were  based  on  accepted  HF  knowledge,  were 
conceptually  validated  by  SMEs,  or  arose  out  of  the  requirements  analyses.  They  include 

•  Reduce  auditory  communication  load  (particularly  at  ORO  and  SWC  positions). 

•  Provide  visual  aids  for  improved  situation  awareness  and  attention  switching  at  all 
positions. 

•  Provide  visual  aids  for  improved  management  of  the  Ops  Rooms  (ORO). 

•  Integrate  all  tabular  information  on  a  contact  in  a  single  place  available  to  all  positions 
(including:  track  number,  bearing  +  trend,  range  +  trend,  course  +  change,  speed  +  change, 
length  (if  ship),  time  held,  current  ID,  recognition  (TG,  ship  and  local),  evidence 
supporting  recognition  assessment. 

•  Provide  critical  contact  data  for  a  track  on  the  situation  display  (e.g.  right  click  or  hover 
brings  up  track  data  block). 

•  Provide  a  visual  aid  that  presents  information  concerning  potential  associations  between 
EW  and  platform  tracks  (e.g.  association  matrix). 

•  Provide  visual  aids  for  management  of  tracks  and  the  tactical  picture  (ORO,  SWC)  - 
including  a  single  source  to  see  all  relevant  information  on  all  tracks,  with  sorting  and 
prioritisation  capability. 
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•  Provide  the  ability  to  predefine  contacts,  areas  and  conditions  of  interest  so  that 
“automated”  processes  can  provide  alerts  or  indicators  when  conditions  are  met. 

•  Provide  visual  aids  on  track  data  displays  to  allow  operators  to  quickly  observe  the  status 
of  pre-defmed  alerts  to  conditions  or  contacts/tracks  of  interest. 

•  Provide  a  consistent  and  common  OMI  design  for  all  displays  that  is  consistent  with 
operators’  existing  mental  models  for  computer  systems  (e.g.  “MS-Windows” 
functionality). 

It  should  be  noted  that  implementation  of  many  of  these  will  have  the  side  benefit  of  reducing  the 
auditory  communication  load. 

Examples  of  possible  implementations  of  some  of  the  above  recommendations  are  provided  in 
Keeble  et.  al.  (2007) 

9.4  Unanswered  questions  and  future  research 

9.4.1  Symbology  for  tactical  displays 

The  research  reviewed  in  Annex  A  along  with  that  carried  out  by  DRDC-Toronto  (McFadden, 

Jeon,  Li,  and  Minniti  2007a)  point  out  some  fundamental  problems  with  existing  and  proposed 
standards  for  symbology.  Further  research  is  required  to  better  understand  (a)  what  critical 
elements  of  the  tactical  picture  need  to  be  represented  on  tactical  displays,  and  (b)  what  tactically 
critical  contact  attributes  need  to  be  represented  in  symbology  and  (c)  what  form  of  encoding  and 
representation  is  best  for  displaying  such  information  (taking  into  account  tactical  context,  overlays 
and  clutter). 

9.4.2  Alerts  and  indicators 

Although  the  project  did  not  specifically  examine  the  issue  of  the  design  of  alerts  and  indicators, 
there  were  several  instances  when  this  issue  was  raised  by  operators  during  the  course  of  interviews 
concerning  tactical  picture  compilation  and  situation  awareness.  Based  upon  such  comments  and 
informal  heuristic  evaluations  of  the  present  CCS  design,  we  believe  that  empirical  studies  need  to 
be  conducted  on  the  best  way  to  implement  alerts  and  other  situation  awareness  aids  and  indicators. 
In  particular,  the  planning  of  the  alert/indicator  hierarchical  scheme  needs  to  be  done  in  a 
disciplined  manner  to  avoid  replicating  existing  alert  problems  with  the  CCS.  Attention  needs  to 
be  given  as  to  how  alerts  should  be  divided  (or  replicated)  between  data  and  tactical  windows.  The 
proposed  scheme  for  indicators  and  alerts  in  Keeble  et  al.  (2007)  is  but  one  possibility  among  many 
which  should  be  empirically  evaluated. 

9.4.3  Representation  of  certainty  and  other  information 

In  the  context  of  MSDF  and  decision  support  algorithms,  the  system  maintains  internally  a 
significant  level  of  detail  on  contact  attributes  and  generates  propositions  concerning  recognition 
state  (and  possibly  identity).  The  issue  of  how  much  of  this  underlying  information  (and  in  what 
format)  should  be  presented  to  the  operator  has  been  touched  upon  in  this  project,  but  requires 
further  analysis  and  investigation.  In  particular,  further  research  is  needed  to  determine  the  best 
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approach  to  representing  confidence  of  information  (used  for  decision  support)  in  a  manner  that  is 
intuitive  to  operators  and  contextually  relevant.  Some  of  the  preliminary  ideas  developed  under 
COMDAT  (McFadden,  Li,  and  Trinh  2007b;  Unger  Campbell  and  Baker  2003)  require  further 
exploration  and  empirical  assessment. 
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10  Conclusions 


This  report  has  provided  a  summary  of  the  HF  work  carried  out  under  COMDAT  along  with 
related  material  that  informed  and  supplemented  this  work,  some  lessons  learned,  and 
recommendations  for  future  work.  The  major  F1F  outcomes  of  the  TDP  were  improved  methods, 
tools,  and  knowledge  about: 

•  cognitive  and  functional  task  analyses  to  defined  the  information  requirements  of  the  major 
Ops  Room  positions  on  Flalifax  Class  ships, 

•  the  development  of  human- in-the-loop  measures  of  performance  to  define  the  current 
performance  in  the  Ops  Room  and  evaluate  new  technology  including  MSDF, 

•  the  design  of  human  computer  interfaces  in  naval  command  and  control  systems  to  better 
meet  the  goals,  information  requirements,  situation  awareness  and  decision  making  of 
individual  operators, 

•  the  design  of  a  concept  of  operations  for  the  implementation  of  MSDF  into  tactical  picture 
compilation,  and 

•  the  design  and  conduct  of  the  evaluation  of  operator  performance  on  existing  and  new 
technologies. 

The  output  from  the  COMDAT  TDP  has  already  informed  the  mid-life  upgrade  of  the  Flalifax 
Class  Ops  Room  and  is  expected  to  define  the  F1F  involvement  in  future  naval  acquisition  projects. 
The  results  and  the  recommendations  for  further  work  are  also  being  passed  on  to  other  TDPs  and 
naval  applied  research  projects. 
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ANNEX  A  -  Human  factors  studies  of 
relevance  to  COMDAT 


The  Technical  Cooperation  Programme  (TTCP),  Human  Resources  and  Performance  Group 
(HUM)  Technical  Panel  9  (TP9):  Human  Systems  Integration  for  Naval  Systems  is  responsible  for 
conducting  collaborative  research  of  mutual  benefit  to  TTCP  nations  aimed  at  integrating  human 
factors  into  the  design  of  safe,  manpower  efficient  and  cost-effective  systems  for  current  and  future 
naval  platforms.  Collaboration  under  TP9  is  carried  out  in  three  main  areas: 

•  Technology  drivers:  evaluating  the  application  and  implications  of  new  technology  for 
naval  systems; 

•  Acquisition  process:  the  insertion  of  HF  into  the  acquisition  process  for  naval  systems; 

•  Tools  for  optimising  naval  platform  manning  requirements. 

Over  the  past  several  years,  research  conducted  jointly  and  separately  by  member  nations  in  each 
of  these  areas  has  been  made  available  to  all  members.  In  particular,  the  research  on  technology 
drivers,  especially  the  research  carried  out  under  the  manning  affordability  project,  proved  of 
immense  value  to  COMDAT.  In  order  to  provide  a  better  understanding  of  the  conclusions  and 
recommendation  put  forward  in  the  various  COMDAT  reports,  this  Annex  provides  a  summary  of 
the  most  relevant  research  provided  under  the  auspices  of  HUM  TP9  and  its  relevance. 

A.1  Operator  Machine  Interface  (OMI)  studies 

The  requirement  in  COMDAT  to  consider  human  factors  issues  in  the  design  of  the  OMI  for  the 
Halifax  Class  Command  and  Control  System  (CCS)  echoed  similar  requirements  in  other  TTCP 
nations.  In  fact,  the  United  States  had  already  conducted  a  series  of  studies  aimed  at  producing 
OMI  design  guidelines  to  support  the  development  of  a  more  effective  and  efficient  interface  for 
the  Aegis  Combat  Information  Center  (CIC)  (Osga  1995).  To  a  large  extent  the  goals  for  the  OMI 
work  under  COMDAT  were  similar,  although  more  modest,  to  those  of  the  Aegis  project. 

However,  the  Aegis  project  followed  a  more  systematic  human  factors  process  starting  with  an 
analysis  of  the  operator’s  tasks.  The  individual  tasks  were  further  analysed  to  determine  generic 
OMI  procedures  and  the  steps  required  to  carry  out  these  procedures.  This  knowledge  was  used  to 
develop  design  concepts  that  supported  efficient  completion  of  these  procedures.  Many  of  the 
concepts  reflect  good  human  factors  practice,  but  they  are  especially  critical  for  the  efficient 
execution  of  time-critical  tasks  carried  out  in  an  Operations  Room.  They  include: 

•  minimize  cursor  movement, 

•  eliminate  or  reduce  unnecessary  visual  shifts  between  alternate  displays, 

•  reduce  nonproductive  shifts  between  input  devices, 

•  efficiently  use  left  and  right  hands  while  resting  arms  comfortably, 

•  use  a  broad  and  shallow  menu  structure, 

•  increase  information  density  and  segmentation  per  display, 
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•  promote  “automatic”  performance  and  sensory  modality  sharing, 

•  provide  constant  visual  feedback  on  task  progress, 

•  reduce  redundant  information  on  separate  displays  and  co-locate  related  information  on  a 
display, 

•  use  “pop-up”  or  “dialog”  windows  to  allow  task  completion  on  a  single  display, 

•  design  flexibility  to  account  for  users  with  varying  system  experience. 

For  more  detailed  discussion  on  each  of  these  design  concepts,  the  reader  is  referred  to  Osga, 
(1995).  Many  of  them  are  consistent  with  the  COMDAT  OMI  Style  Guide. 

To  evaluate  their  utility,  the  Aegis  CIC  console  was  redesigned  to  be  consistent  with  these  concepts 
and  task  completion  times  on  the  original  and  redesigned  consoles  were  compared  using  a  Micro- 
Saint  model  (Osga  1989).  The  model  was  populated  with  actual  response  times  for  the  actions 
required  to  complete  procedures  on  the  existing  console  and  estimated  times  for  the  actions 
required  to  complete  the  same  procedures  on  the  redesigned  console.  On  average,  they  found  a 
62%  reduction  in  completion  times  with  the  redesigned  console.  Since,  the  model  did  not  assume 
improvements  in  times  to  absorb  information  because  the  presentation  format  was  improved,  this 
finding  was  probably  conservative. 

In  order  to  evaluate  specific  technologies  for  implementing  the  design  concepts,  a  part  task 
simulator,  the  Navy  Advanced  Information  Management  Evaluation  System  (NAIMES),  was 
developed.  Initially  the  simulator  was  used  to  evaluate  different  input  devices,  menu 
configurations,  and  console  layouts.  With  it,  operators  could  carry  out  typical  tasks  such  as  hooking 
tracks,  adding  tactical  symbols  to  the  display,  and  carrying  out  a  multitask  sequence  that  included 
hooking  tracks,  accessing  menus,  and  executing  functions.  Details  of  the  results  can  be  found  in 
Osga  (1995).  The  specific  findings  are  probably  of  limited  relevance,  since  they  involved  1990’s 
technology,  but  the  analyses  of  the  suitability  of  different  types  of  input  devices,  menu  formats,  and 
console  layouts  for  a  CCS  type  display  remain  relevant.  The  work  also  illustrates  the  potential  of 
using  a  relatively  simple  simulation  to  evaluate  OMI  concepts  instead  of  a  full  scale  prototype. 

Such  an  approach  ensures  that  most  OMI  design  issues  are  handled  early  in  the  design  process  and 
reduces  the  likelihood  that  evaluation  of  a  final  prototype  will  be  negatively  impacted  by  simple 
OMI  design  errors. 


A.1.1  Studies  on  symbology 

The  results  of  these  initial  evaluations  were  used  to  provide  short  term  recommendations  for 
improving  the  OMI  of  the  Aegis  CIC.  Following  the  completion  of  this  work,  NAIMES  was 
redesigned  to  allow  the  investigation  of  more  innovative  technologies  that  could  be  introduced  in 
the  medium  to  long  term.  A  major  part  of  this  work  was  the  development  of  an  improved 
symbology  set.  As  with  the  current  Halifax  Class  CCS,  the  Aegis  CIC  consoles  employed  the  Navy 
Tactical  Data  System  (NTDS)  symbols  set.  In  the  case  of  the  Aegis  console,  only  monochrome 
NTDS  symbols  were  used.  In  addition  to  the  lack  of  colour  coding,  the  NTDS  symbols  provided 
operators  with  minimal  information  about  the  track  other  than  position,  warfare  area,  and  standard 
identification.  Thus,  a  series  of  studies  were  conducted  to  evaluate  an  alternative  symbol  set  that 
was  being  developed  by  a  North  Atlantic  Treaty  Organisation  (NATO)  working  group  (North 
Atlantic  Treaty  Organisation  1992).  In  general  the  results  of  these  studies  supported  the  use  of  solid 
colour  coded  shapes  that  could  be  annotated  with  icons  that  provided  supplemental  platform 
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information  about  tracks  (Kirkpatrick  et  al.  1992a;  Kirkpatrick  et  al.  1992b; 
NAVSPAWARSYSCOM  1991).  These  studies,  which  focused  primarily  on  air  and  surface 
symbology,  formed  the  basis  for  the  symbology  studies  carried  out  under  COMDAT  (McFadden  et 
al.  2007a).  A  more  detailed  summary  of  the  results  and  their  implications  for  the  design  of 
HALIFAX  Class  tactical  displays  are  available  in  that  paper. 

The  availability  of  multiple  symbol  sets  in  the  NAIMES  environment  supported  the  evaluation  of 
an  innovative  method  for  reducing  clutter  and  supporting  information  processing  -  variable  coded 
symbology  (Osga  and  Keating  1994).  The  idea  was  to  use  different  symbol  sets  (and  other 
concepts),  in  association  with  user  defined  filters,  to  increase  the  salience  of  task  relevant  tracks 
and  decrease  the  salience  of  other  tracks.  Osga  and  Keating  provided  operators  with  access  to  six 
different  symbol  sets  -  the  standard  NATO  symbology,  the  basic  NATO  symbology  (no 
annotation),  colour-coded  NTDS  symbols,  white  NTDS  symbols,  grey  NTDS  symbols,  coloured, 
and  black  dots.  Operators  associated  the  different  types  of  tracks  that  met  the  criterion  of  their 
filters  with  the  different  symbol  sets.  They  could  also  select  amplifying  information  for  specific 
types  of  tracks  For  example,  they  could  have  two  sets  of  tracks  (hostile  fighters  and  warships) 
coded  NATO  full  and  have  one  with  leaders  and  one  without.  The  initial  study  looked  at  the 
number  of  filters  and  levels  within  a  filter  that  operators  would  generate.  Typically,  operators 
created  between  2  and  4  filters  with  between  2  and  1 7  levels  for  each.  Overall,  23  unique  filters 
were  created.  Next,  participants  used  their  filters  in  monitoring  a  dynamic  scenario.  On  the  whole, 
the  response  was  positive.  Operators  liked  the  concept  and  thought  it  had  merit  for  reducing  the 
complexity  of  a  tactical  display.  Unfortunately,  there  were  no  comparisons  of  performance  or 
situation  awareness  across  the  different  sets  of  filters.  One  concern  is  that  this  type  of  manipulation 
could  lead  to  lower  awareness  of  new  tracks. 

The  absence  of  performance  assessment  was  partially  overcome  with  two  later  studies  by  Nugent 
(1996)  and  Van  Orden,  Nugent,  La  Fleur,  and  Moncho  (1999).  Both  studies  compared  performance 
with  variable  coded  symbology  (VCS)  and  conventional  symbology  on  a  visual  search  task.  Three 
symbols  sets,  monochrome  and  colour  coded  NTDS  symbols  and  colour  filled  NATO  tactical 
symbols,  were  used  in  the  variable  coded  symbology  condition  and  colour-coded  NTDS  symbols 
were  used  in  the  control  or  conventional  display  condition.  There  were  three  VCS  conditions  -  (1) 
colour  NTDS  for  prominent  and  monochrome  NTDS  for  receded,  (2)  NATO  symbols  for 
prominent  and  colour  NTDS  for  receded,  and  (3)  NATO  for  high,  colour  NTDS  for  moderate,  and 
monochrome  NTDS  for  low  prominence  targets.  The  operator’s  task  was  to  locate,  select,  and  enter 
as  many  target  symbols  on  a  simulated  tactical  display  as  possible.  The  Nugent  study  used 
throughput  (a  combination  of  accuracy  and  response  time)  and  confirmation  time  as  the  measures 
of  performance.  The  second  study  (Van  Orden  et  al.  1999)  used  response  time  per  target.  The 
results  were  similar  in  both  studies.  Overall,  the  use  of  variable  coded  symbology  did  not  have 
either  a  positive  or  negative  impact  on  performance.  Instead,  performance  within  each  condition 
tended  to  be  a  function  of  the  target’s  symbol  set.  In  particular,  performance  was  poorest  when 
participants  had  to  search  for  targets  coded  using  monochrome  NTDS  symbols  in  VCS  1  and  best 
when  the  target  symbol  was  coded  using  the  NATO  symbology  in  VCS3.  The  difference  in  results 
could  not  be  attributed  to  the  number  of  symbols  displayed  in  each  symbol  set.  The  Van  Orden  et  al 
(1999)  study  also  collected  eye  movement  data  recording  the  number  of  fixations  per  target.  That 
data  indicated  that  poor  performance  was  linked  to  the  presence  of  both  white  and  grey  symbols  on 
the  same  display.  Thus,  they  recommended  that  variable  coding  combinations  should  not  include 
both  grey  and  colour-coded  NTDS  symbols.  These  results  support  the  need  for  more  systematic 
study  of  the  factors  that  affect  discrimination  of  symbols  in  complex  displays  (McFadden  et  al. 
2007a). 
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One  of  the  potential  limitations  of  using  filled  symbols  is  that  some  tracks  are  more  likely  to  be 
partially  or  totally  obscured  on  cluttered  displays.  Nugent  (1994)  and  Nugent,  Keating,  and 
Campbell  (1995)  investigated  this  possibility  along  with  two  potential  tools  for  overcoming  it.  The 
two  tools  were  a  pop-up  window  that  listed  all  tracks  in  a  small  area  surrounding  the  coordinate 
position  selected  on  the  display  and  a  click  tool  that  brought  each  symbol  in  the  group  to  the 
forefront  on  successive  mouse  clicks  in  the  area  containing  a  cluster  of  symbols.  The  NATO 
symbol  set  was  compared  against  a  colour-coded  NTDS  symbols  set  that  had  been  amplified  to 
provide  a  similar  level  of  information  to  the  annotated  NATO  symbols.  Operators  searched  for 
target  symbols  under  a  no  overlap,  partial  overlap,  total  overlap  and  an  “on  top”  condition.  In  the 
latter,  the  target  symbol  was  always  positioned  on  top  of  two  or  more  adjacent  symbols.  They  were 
expected  to  search  for  the  target  symbol,  using  the  selection  tool  and  hook  potential  targets 
symbols.  Performance  was  faster  and  more  accurate  with  the  NATO  symbols  in  the  no,  partial  and 
“on  top”  overlap  conditions,  but  there  were  fewer  time-outs  with  the  NTDS  symbols  (failure  to  find 
all  the  targets  in  a  two  minute  period).  Moreover,  a  more  detailed  analysis  indicated  that  the 
superiority  of  the  NATO  symbols  occurred  primarily  in  the  “on  top  condition.  In  the  total  overlap 
condition,  performance  was  more  accurate  with  the  NATO  symbols,  but  it  was  significantly  faster 
and  there  was  less  use  of  the  selection  tools  with  the  NTDS  symbols.  The  faster  response  time  was 
primarily  due  to  the  larger  number  of  time-outs  with  the  NATO  symbols.  There  was  no  difference 
in  performance  between  the  tools  except  that  operators  using  the  click  tool  made  less  use  of  it  and 
had  twice  as  many  time-outs.  This  difference  was  attributed  to  the  ability  to  review  all  symbols  in  a 
group  simultaneously  with  the  pop-up  tool. 

The  results  of  the  above  studies  formed  the  basis  for  the  initial  interface  for  the  Multi  Modal  Work 
Station  (MMWS)  which  was  developed  to  explore  more  advanced  concepts  in  interface  design  in 
support  of  the  manning  affordability  project. 


A.1 .2  Support  for  the  management  of  tasks 

In  a  complex  environment  such  as  an  operations  room,  operators  must  complete  a  wide  range  of 
tasks  and  task  sequences  accurately  and  quickly.  There  can  be  a  heavy  workload  associated  with 
the  planning  and  time  management  of  these  tasks.  In  an  attempt  to  reduce  this  type  of  workload, 
concepts  to  support  task  management  were  investigated  as  part  of  the  MMWS  project.  Initially,  a 
set  of  task  characteristics  and  the  design  requirements  for  these  characteristics  were  defined.  For 
example  if  a  characteristic  of  a  task  is  that  it  has  defined  schedule  then  that  schedule  should  be 
made  available  to  the  operator  (Osga  2000).  Design  requirements  were  associated  with  these 
characteristics  and  in  turn  these  were  matched  with  a  variety  of  user-interface  aids  to  support  tasks 
with  those  characteristics.  Thus,  the  system  might  provide  a  display  showing  a  summary  of  the 
steps  associated  with  the  task  of  prosecuting  a  hostile  target  and  by  monitoring  the  system  status 
alert  the  operator  when  the  next  step  needed  to  be  carried  out.  Tasks  were  looked  at  from  the 
perspective  of  their  impact  on  cognitive  performance.  Tasks  that  were  skill-  or  rule-based  were 
good  candidates  for  automation  as  were  information  seeking  tasks. 

The  utility  of  one  possible  type  of  task  management  display  was  evaluated  empirically  by  St.  John 
and  Osga  (St.  John  and  Osga  1999)  using  a  generic  task.  A  simulation  was  developed  in  which 
participants  supervised  a  group  of  friends  (tasks)  getting  ready  for  a  trip  to  a  shopping  mall.  The 
supervisor  had  to  monitor  six  concurrent  tasks  and  evaluate  the  effect  of  problems  that  arose  on 
deadlines.  They  were  provided  with  a  display  that  listed  a  set  of  tasks  and  subtasks,  including  their 
start  time  and  completion  time,  that  needed  to  be  completed  in  order  for  the  supervisor  to  meet 
his/her  goal.  A  simple  Gantt  chart  layout  was  used  that  mapped  the  tasks  and  subtasks  against 
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timelines.  The  supervisor  received  alerts  about  the  tasks.  The  goal  of  the  experiment  was  to  see  if 
the  task  manager  display  could  improve  the  evaluation  of  alert  messages.  A  second  experiment 
added  a  decision  support  tool  that  ranked  the  alerts  into  one  of  four  priorities  and  informed  the 
supervisor  in  one  of  two  ways  about  the  priority  of  the  alert.  In  general,  the  results  showed  the 
value  of  simple  displays  that  reduce  memory  load  associated  with  remembering  the  spatial- 
temporal  order  of  tasks  and,  in  the  second  experiment,  that  reduce  the  workload  associated  with 
determining  which  of  multiple  tasks  need  to  be  addressed  first. 

An  important  issue  in  the  design  of  task  management  support  is  how  to  integrate  it  into  the  overall 
interface.  User  evaluation  of  a  prototype  task  management  display  implemented  in  the  MMWS 
suggested  that  the  use  of  time  and  display  scrolling  were  not  beneficial  (Osga  2000). 


A.1 .3  Evaluation  of  3-dimensional  display  concepts 

Advances  in  graphic  engines  have  led  to  the  development  and  investigation  of  more  advanced 
concepts  in  symbology  such  as  the  use  of  3 -dimensional  (3-D)  displays  and  icons  (Smallman, 
Schiller,  and  Mitchell  1999;  Smallman,  Oonk,  St.  John,  and  Cowan  2000a;  Smallman,  Schiller,  and 
Cowen  2000b;  Smallman,  St.  John,  Oonk,  and  Cowen  2000c;  Smallman,  Oonk,  St.  John,  and 
Cowen  2001a).  An  initial  study  (  Smallman  et  al.  1999)  looked  at  the  utility  of  a  3-D  display  for 
improving  rapid  situation  awareness  (SA),  where  SA  was  defined  as  the  percentage  of  correctly 
answered  questions  about  specific  display  attributes.  Participants  monitored  a  tactical  scenario 
presented  as  a  3-D  display  with  realistic  icons,  a  2-dimensional  (2-D)  display  with  realistic  icons, 
or  a  2-D  display  with  conventional  symbols.  In  this  case,  MIL-STD-2525B  symbology 
(Department  of  Defense  1999)  was  used.  Every  thirty  seconds,  participants  were  probed  about  the 
spatial  and  identity  attributes  of  four  of  the  targets.  Initially  SA  was  best  with  the  2-D  display  with 
conventional  symbols,  but  over  time,  it  increased  in  the  other  two  conditions  until  it  was  similar  in 
all  three  conditions.  In  terms  of  the  individual  attributes,  spatial  attributes,  especially  altitude  and 
attitude,  were  recalled  better  with  the  3-D  icons  in  the  3-D  display  while  identity  and  platform 
information  was  more  accurate  with  the  2-D  symbols. 

One  problem  with  3-D  perspective  displays  is  the  inherent  ambiguities  of  perspective  projection 
and  the  lack  of  depth  cues  making  it  difficult  to  accurately  identify  the  position  of  tracks. 

Smallman,  Schiller,  and  Cowan  (2000b)  investigated  three  techniques  for  overcoming  this  problem 
-  drop  lines  (a  line  that  extended  from  the  centre  of  the  icon  to  the  ground  plane)  and  drop  shadows 
(a  shadow  of  the  icon  appears  directly  under  the  symbol  on  the  ground  plane)  and  scaling  of  track 
symbol  size.  The  first  two  techniques  are  designed  to  unambiguously  locate  the  position  of  the 
icon.  In  addition  the  length  of  the  line  and  the  distance  between  the  icon  and  its  shadow  provides  an 
estimate  of  altitude.  Participants  were  shown  seven  versions  of  a  tactical  picture:  six  3-D  views 
and  one  2-D  view.  In  three  of  the  3-D  views,  the  icons  were  all  the  same  size  and  in  the  other  three 
they  varied  continuously  with  distance.  Within  each  set  of  three,  one  scene  used  drop  lines,  a 
second  used  drop  shadows  and  a  third  used  nothing.  Participants  were  asked  to  reconstruct  each 
display  by  placing  pins  on  a  colour  printout  of  the  background  topography  where  they  thought  the 
track  icons  appeared  on  the  display.  Misalignment  was  significantly  less  with  the  use  of  both  drop 
lines  and  drop  shadows.  However,  changing  aircraft  size  as  a  function  of  distance  was  only 
beneficial  when  it  was  the  only  depth  cue.  There  was  no  benefit  to  using  the  combined  cues.  In 
fact,  misalignments  were  significantly  greater  with  the  use  of  size  change  and  drop  lines. 

Other  studies  by  the  same  group,  (Smallman  et  al.  2000a;  Smallman  et  al.  2000c;  Smallman  et  al. 
2001a)  focused  on  the  relative  advantage  of  3-D  icons  and  2-D  symbols  for  representing  track 
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attribute  information.  Initially  (Smallman  et  al.  2000a;  Smallman  et  al.  2000c),  participants  were 
shown  the  icons  or  symbols  one  at  a  time  and  asked  to  name  them  (i.e.  the  platform)  as  quickly  as 
possible.  Platform  identification  performance  was  consistently  better  with  the  2-D  symbols  than 
with  the  3-D  icons.  The  authors  attributed  this  finding  to  the  fact  that  2-D  symbols  can  be  created 
with  high  discriminability.  With  the  realistic  3-D  icons,  it  is  often  necessary  for  operators  to 
discriminate  subtly  different  looking  platforms. 

The  final  study  (Smallman  et  al.  2001a)  examined  a  broader  range  of  attributes,  represented  by  3-D 
icons,  2-D  icons,  or  2-D  symbols,  using  a  visual  search  task.  The  symbols  and  icons  were  presented 
on  either  a  2-D  or  3-D  display.  In  general,  tracks  with  the  specified  attribute  (e.g.  platform,  altitude 
heading),  were  located  faster  when  symbols  were  used  rather  than  icons  and  with  a  2-D  display  as 
opposed  to  a  3-D  display.  However,  there  were  exceptions.  Searching  for  heading  and  speed  was 
faster  with  3-D  icons,  but  only  on  the  2-D  display.  The  reason  for  the  latter  is  that  heading 
(direction)  and  attitude  (ascending  etc)  are  confounded  with  3-D  icons  seen  in  a  3-D  perspective 
view.  The  heading  advantage  for  icons  was  attributed  to  the  fact  that  the  icon  itself  indicated 
direction.  With  the  symbols,  the  symbol  heading  does  not  change  but  a  short  line  (called  a  leader) 
attached  to  the  symbol  indicates  heading.  The  speed  advantage  was  attributed  to  the  fact  that  speed 
was  linked  to  platform  category  (air  versus  surface/subsurface)  and  these  two  categories  were  more 
discriminable  when  imaged  as  icons  than  as  symbols.  The  overall  conclusion  of  the  authors  was 
that  realistic  coding  (3-D  icons  in  a  3-D  perspective  view  display)  puts  an  extremely  high  burden 
on  the  visual  attribute  of  shape.  With  symbols,  it  is  possible  to  map  attributes  onto  different 
dimensions  of  the  symbol  and  to  make  differences  along  a  dimension  highly  discriminable. 


A.1 .4  Representation  of  track  heading 

In  an  attempt  to  improve  the  discrimination  of  category  and  heading  with  symbols,  Smallman,  St. 
John,  Oonk,  and  Cowen  (2001b)  proposed  a  set  of  hybrid  symbols,  which  they  called 
“Symbicons”.  These  were  simplified  versions  of  a  plane  and  boat  and  the  direction  the  Symbicon 
pointed  indicated  the  direction  the  track  was  moving.  Using  the  same  visual  search  task  described 
in  the  previous  paragraph,  they  found  that  performance  with  the  Symbicons  was  equal  to  or  better 
than  performance  with  the  symbols  on  affiliation  and  platform  and  to  the  icons  on  heading  and 
platform  category.  Thus,  Symbicons  successfully  combined  the  best  characteristics  of  symbols  and 
icons. 

A  follow-on  study  (Keillor,  Thompson,  Smallman,  and  Cowen  2003)  supported  the  use  of 
Symbicons  for  detecting  a  change  in  heading.  In  it,  participants  search  for  a  change  in  heading  on 
a  tactical  display  where  track  attributes  were  imaged  using  either  Symbicons  or  2525B  symbology. 
Participants  located  the  heading  change  more  quickly  when  Symbicons  were  used. 

This  study  used  a  change-blindness  paradigm.  Change  blindness  refer  to  the  phenomenon  that 
humans  are  often  unable  to  detect  major  changes  in  objects  in  two  successively  presented  scenes 
(DiVita  and  Nugent  2000).  Typically,  two  images  of  a  scene  that  differ  only  in  one  aspect,  (e.g.  the 
heading  of  a  specific  track)  are  alternated.  A  blank  screen  is  presented  briefly  after  each 
presentation  of  one  of  the  scene  images.  These  three  images  are  presented  continuously  until  the 
participant  locates  the  change.  DiVita  and  Nugent  (2000)  investigated  the  use  of  change-blindness 
for  evaluating  graphical  interfaces  and  showed  that  it  occurred  even  in  relatively  low  workload 
conditions  when  people  are  focused  on  monitoring  a  display  for  change.  The  Symbicon  study  by 
Keillor  et  al  (2003)  demonstrated  that  the  paradigm  could  be  used  successfully  to  evaluate  design 
concepts.  It  has  the  added  advantage,  unlike  most  visual  search  tasks,  that  participants  are  not 
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searching  for  a  specified  object  which  is  closer  to  the  actual  task  of  operators  monitoring  tactical 
displays. 

One  potential  limitation  of  Symbicons,  in  respect  to  their  use  in  a  Halifax  class  Ops  Room,  was  that 
the  same  shape  was  used  for  both  surface  and  underwater  tracks.  Another  possible  limitation  is  that 
there  is  no  redundant  coding  of  affiliation  and  shape.  However,  our  own  research  suggests  that  the 
blocky  shapes  used  in  M1L-STD  2525B  are  not  particularly  discriminable  (unless  redundantly 
colour  coded)  (McFadden  et  al.  2007a).  Another  issue  is  the  broader  impact  of  rotating  the  whole 
symbol.  While  rotating  the  shape  improves  recognition  of  heading,  it  could  reduce  recognition  of 
the  platform  category  in  a  more  complex  environment  with  a  large  number  of  unique  shapes.  In 
general  these  studies  show  the  potential  value  and  limitations  of  symbology  for  encoding 
information  and  point  to  the  need  for  more  systematic  research  on  design  concepts  for 
multidimensional  symbols  on  complex  display. 


A.1.5  Situation  awareness  for  changes  in  the  tactical  picture 

Another  approach  to  ameliorating  the  effects  of  change  blindness  has  been  explored  by  Smallman 
and  St  John  (2003)  in  their  investigation  of  a  tool  that  they  called  CHEX  (Change  History  Explicit). 
The  tool  monitored  the  output  of  a  simulated  tactical  display  and  logged  significant  changes  in 
critical  tracks  in  a  reconfigurable  table  displayed  beside  the  tactical  display.  Entries  in  the  table 
were  dynamically  linked  to  the  tracks  on  the  tactical  display.  They  found  that  CHEX  was  very 
helpful  in  maintaining  situation  awareness  when  monitoring  the  display  and  for  recovering 
situation  awareness  after  an  interruption  especially  as  track  density  increased.  Performance  with 
CHEX  was  compared  against  conditions  with  no  support,  a  static,  unlinked  table,  and  the  static 
table  with  the  critical  tracks  circled  in  red.  A  second  experiment  compared  CHEX  with  the  ability 
to  replay  interrupted  periods  at  high  speed.  Again,  the  CHEX  tool  proved  superior.  Further  research 
would  be  required  to  determine  the  overall  efficacy  of  such  a  tool  in  an  operational  setting,  since 
many  operators  are  loathed  to  take  their  eyes  off  the  tactical  screen.  On  the  other  hand,  some 
operators  may  focus  more  on  the  output  of  the  tool  instead  of  the  tactical  display.  This  could  affect 
their  comprehension  of  the  overall  tactical  picture. 


A.1.6  Automation 

Tools  such  as  CHEX  reflect  the  use  of  underlying  automation  to  carry  out  monitoring  tasks  that 
humans  are  known  to  be  notoriously  poor  at  (Parasuraman  1986).  Human  use  of  automation  has 
been  extensively  studied  particularly  in  the  cockpit.  A  common  finding  is  that  human-system 
performance  in  systems  with  extensive  automation  tends  to  be  sub-optimal.  People  are  either  over 
reliant  on  automation  or  ignore  it  (Endsley  1996).  Since  the  MMWS  included  substantive 
automation,  one  study  (St.  John,  Oonk,  and  Osga  2000)  examined  the  impact  of  reducing  the 
reliability  of  the  automation  that  prioritized  alerts  in  a  task  management  system  such  as  the  one 
implemented  in  the  MMWS.  The  experiment  compared  100%  accuracy  against  75%  accuracy.  The 
reduction  in  accuracy  was  due  to  a  switching  in  priority  of  two  of  the  tasks.  In  one  case,  the  human 
was  informed  that  the  automation  had  not  been  informed  of  the  switch  and  in  the  other  they  were 
just  told  that  the  reliability  of  the  automation  had  been  reduced.  There  were  also  two  manual 
conditions,  one  with  the  original  priority  and  one  in  which  the  priority  was  switched.  In  the 
informed  and  manual  conditions,  the  participants  were  informed  verbally  that  the  priority  had 
switched  but  the  display  was  not  updated.  In  the  uninformed  condition,  the  new  priority  was  shown 
on  the  display.  The  two  tasks,  whose  priority  was  switched,  did  not  occur  until  half  way  through 
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the  scenario.  Thus,  in  the  first  half  of  the  scenario  the  automation  worked  perfectly.  However,  the 
performance  of  the  informed  group  was  still  poorer  than  that  of  the  100%  reliable  condition 
although  it  was  better  than  the  uniformed  condition  in  terms  of  number  of  errors.  In  the  second 
half,  all  groups  except  the  reliable  automation  group  performed  similarly.  Of  interest  is  the  fact 
that,  in  the  informed  group,  8  of  the  1 1  participants  relied  on  the  automation  and  the  others  handled 
the  task  manually  during  the  first  half.  The  results  suggest  that  being  aware  of  the  conditions  under 
which  automation  is  likely  to  fail  can  lead  to  more  effective  use  of  it,  but  not  uniformly.  Also  some 
people  appear  to  be  able  to  make  use  of  unreliable  automation  even  when  they  do  not  know  why  it 
might  fail. 

The  results  of  this  study  are  typical  of  studies  on  human  use  of  automation.  Since  future  naval 
systems  will  include  substantial  automation,  it  is  important  to  understand  how  to  design  systems 
employing  automation  in  order  to  optimize  human-system  performance.  As  well  as  providing 
useful  information,  this  work  illustrates  the  use  of  part  task  experiments  to  evaluate  specific  design 
concepts.  Such  studies  allow  the  collection  of  larger  and  more  rigorous  data  sets  than  would  be 
possible  using  existing  systems  and  Navy  SMEs  while  still  providing  directly  relevant  knowledge 
that  can  be  used  in  the  design  of  a  prototype  system. 

A.2  Performance  measurement 

A  key  aspect  of  the  research  conducted  in  support  of  the  MMWS  was  the  development  of  improved 
methods  and  measures  for  assessing  performance  in  both  part  tasks  and  for  evaluation  of 
prototypes  and  existing  systems.  As  with  the  previous  two  sections,  the  research  discussed  below 
has  supported  and  supplemented  the  research  on  MOPs  conducted  under  COMDAT  and  provided 
significant  insight  into  possible  methodologies  for  measuring  performance  in  complex  naval 
systems. 


A.2.1  Workload 

One  of  the  most  common  measures  of  performance  used  in  assessing  prototype  systems  is 
workload.  It  is  often  assessed  using  subjective  methods  such  as  rating  scales  that  are  presented 
during  a  scenario  (task)  or  after  its  completion  (Gopher  and  Donchin  1986).  One  commonly  used 
scale  for  that  purpose  is  the  NASA  TLX  (Hart  and  Staveland  1988).  While  the  evaluation  of 
workload  using  such  scales  has  been  reasonably  successful,  like  all  subjective  methods,  it  can 
interfere  with  the  tasks  being  carried  out,  and  is  subject  to  participant  bias.  Other  methods,  that  do 
not  have  these  limitations,  include  physiological  and  performance  tests.  However,  they  also  have 
limitations  (Gopher  and  Donchin  1986).  The  MMWS  attempted  to  avoid  the  limitations  of 
individual  measures  by  including  a  real-time  workload  measurement  capability  (Van  Orden  2000) 
that  integrated  a  whole  range  of  measures  designed  to  assess  operator  activity,  environmental  (task) 
characteristics,  and  the  psychophysiological  state  of  the  operator.  The  overall  system  was  to  be 
composed  of  multiple  modules  each  of  which  collected  a  specific  measure.  The  output  of  each 
module  was  fed  into  an  integrator  that  produced  an  overall  assessment.  Unfortunately,  it  was  not 
possible  to  implement  all  of  the  modules  as  some  of  the  methods  required  substantial  development 
to  turn  them  into  real-time  estimates.  Van  Orden  (2000)  described  one  system  to  predict  drowsiness 
and  high  visual  workload  using  monitoring  of  several  measures  of  eye  activity.  He  compared  eye 
activity  in  a  boring  monotonous  task  over  time  and  also  in  tasks  involving  contact  density  and 
found  a  reasonable  correlation  between  contact  density  and  eye  activity  at  the  high  end.  However, 
with  low  contact  density,  there  was  evidence  of  additional  activity  probably  associated  with  other 


Page  88 


Final  COMDAT  Human  Factors  Report  Humansystems  Incorporated 


visual  activity.  This  problem  became  more  evident  when  he  tried  to  use  this  measure  in  complex 
search  and  selection  tasks.  Other  modules  that  have  been  investigated  include  speaking  rate  and 
button  pushing  rate.  The  paper  also  included  a  discussion  of  initial  efforts  to  correlate  some  of  the 
workload  measures  with  task  activity.  The  general  conclusion  was  that  it  will  probably  be  difficult 
to  accurately  estimate  operator  workload  in  any  simple  way.  It  is  necessary  to  take  into  account 
modulation  factors  such  as  operator  experience  and  system  state.  The  conclusion  was  that  a  model 
that  captures  the  relative  importance  of  different  workload  measures  is  likely  to  be  individual  and 
system  specific.  This  suggests  that  developing  a  practical  real-time  capability  for  monitoring 
workload  is  unlikely  unless  it  includes  a  learning  component. 

A.2.2  Evaluating  new  technology 

As  with  the  COMDAT  TDP,  the  manning  affordability  project  included  the  development  of 
methods  and  measures  of  performance  for  assessing  the  impact  of  the  new  technology  developed 
under  it  and  implemented  in  the  Integrated  Combat  Environment  (ICE)  and  the  MMWS  (Freeman, 
Campbell,  and  Hildebrand  2000).  The  goal  was  to  develop  measures  that  could  be  used  in 
evaluating  whether  the  advanced  command  and  control  watchstation,  as  represented  by  the  MMWS 
in  the  ICE  laboratory,  could  be  manned  successfully  by  fewer  operators  than  a  traditional  combat 
information  centre.  The  approach  they  used  was  to  developed  a  scenario  designed  to  allow  certain 
behaviours  to  emerge  and  then  measured  those  behaviours.  The  MOPs  centred  on  the  detect-to- 
engage  cycle  and  included  a  combination  of  subjective  and  objective  measures.  Specific  MOPs 
included: 

•  timeliness  and  accuracy  in  the  detection  and  response  to  tracks, 

•  communication  level  and  content, 

•  situation  awareness,  and 

•  workload. 

The  results  reported  by  Freeman  et  al.  (2000)  were  restricted  to  data  collected  using  an  imbedded 
training  system  in  the  Aegis  Combat  Information  Center  using  the  anti-air  warfare  component  of  a 
standard  watchstanding  team.  They  did  compare  operator  and  observer  estimates  of  workload  over 
high  and  low  workload  parts  of  the  scenario  and  found  that  workload  varied  as  a  function  of  the 
scenario  and  that  the  two  estimates  were  similar.  A  subsequent  paper  (Osga,  Van  Orden, 

Kellmeyer,  and  Campbell  2001)  includes  some  comparisons  of  the  performance  data  on  the  legacy 
system  with  performance  on  the  MMWS.  It  reported  that  the  operators’  situation  awareness 
improved  substantially  and  workload  was  lower  when  using  the  MMWS  as  compared  to  the 
operational  system  even  though  the  latter  was  manned  by  an  eight  person  team  and  the  MMWS  by 
a  four  person  team. 

The  work  reported  in  these  papers  is  very  similar  to  work  carried  out  under  COMDAT  on  the 
development  of  MOPs  (Section  4)  and  the  evaluation  of  performance  on  the  existing  system 
(Section  5)  including  many  of  the  same  MOPs.  Thus,  it  provides  some  validation  for  the  approach 
proposed  under  COMDAT  and  its  potential  utility  for  evaluating  new  technology  such  as  MSDF. 
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A.2.3  Modelling  and  simulation 

An  alternative  to  the  two  types  of  studies  reported  above  is  the  use  of  models  to  predict 
performance  and  workload.  Although  modelling  can  be  as  expensive  to  implement  as  an 
experiment,  it  permits  evaluation  of  a  much  greater  range  of  parameter  values  than  could  ever  be 
accomplished  in  experiments.  The  output  of  requirements  analysis  can  be  used  to  create  models  of 
complex  task  sequences  in  programmes  such  as  the  Integrated  Performance  Modelling 
Environment  (IPME).  The  utility  of  such  models  for  predicting  performance  times  was  also 
investigated  under  the  MMWS  project  (Carolan,  Scott-Nash,  Corker,  and  Kellmeyer  2000).  Several 
different  models  of  four  different  usability  tasks  designed  to  measure  the  impact  of  different  input 
modalities  were  developed  and  run  in  IPME.  The  performance  predictions  of  the  model  were 
compared  against  the  performance  of  people  carrying  out  the  same  tasks.  The  estimates  were  quite 
similar.  They  then  put  the  actual  data  into  the  model  and  compared  the  model’s  predictions  with 
human  performance  data  on  a  more  complex  task  that  utilized  the  original  usability  tasks.  Again 
the  predictions  were  quite  good.  Models  of  the  task  management  functions  and  of  the  full  scenario 
used  in  the  evaluations  discussed  previously  (Osga  et  al.  2001)  were  also  created.  Comparisons 
with  actual  data  should  further  improve  knowledge  about  the  utility  of  modelling  as  part  of  the  tool 
set  available  for  evaluating  design  concepts  for  new  systems.  A  similar  approach  is  proposed  in 
section  5  to  overcome  the  difficulty  of  accessing  Canadian  Navy  training  facilities  and  personnel 
for  HIL  evaluations  of  new  technology.  These  results  suggest  the  approach  has  merit. 


A.2.4  Usability  analysis 

Another  standard  approach  to  evaluating  proposed  systems  is  usability  analysis.  The  use  of  this 
type  of  methodology  during  the  various  evaluations  of  the  design  process  with  the  MMWS  and 
some  lessons  learned  are  described  by  Kellmeyer  and  Osga  (2000).  Initially,  they  trace  out  the 
design  process  and  link  the  process  to  the  type  of  evaluation  and  use  of  SMEs.  The  paper  includes 
some  useful  lessons  learned.  The  process  is  similar  to  that  described  in  section  2. 1  in  that  it 
involves  increasing  fidelity  of  representation  of  the  proposed  design  and  a  move  from  subjective  to 
more  objective  evaluations.  Some  lessons  learned  included: 

•  They  found  that  SMEs  had  trouble  relating  to  a  future  tasking,  the  use  of  higher  levels  of 
automation,  and  new  roles  due  to  crew  reduction.  Thus,  the  initial  evaluations  tended  to 
focus  on  display  layout  and  later  evaluations  used  scenarios  that  involved  current  taskings. 
A  similar  reaction  was  encountered  in  many  of  the  COMDAT  evaluations.  SMEs  could  not 
imagine  that  they  would  not  be  able  to  handle  a  large  increase  in  the  number  of  tracks 
using  the  procedures  that  they  do  now. 

•  They  warned  against  spending  too  much  time  and  effort  on  lower  fidelity  levels  of 
prototyping  (storyboards,  poweipoint  etc).  Only  high  fidelity  prototyping  can  bring  out  the 
complications  of  dynamic  usability.  Their  opinion  was  that  continued  refinement  of  low- 
fidelity  (i.e.  part-task)  interfaces  leads  to  increased  display  complexity.  This  unwanted 
complexity  only  becomes  apparent  when  the  designs  are  tested  within  the  overall  system. 
One  example  is  the  use  of  scrolling  and  timelines  in  the  task  management  display 
discussed  earlier  in  Section  A.  1.2. 

•  They  also  warned  against  putting  extensive  effort  into  development  of  the  interactive 
prototype.  They  found  that  the  interactive  prototype  took  far  too  long  to  build  and  iterate 
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and  in  the  end  became  too  complicated  to  easily  build  on.  They  recommended  focusing  on 
key  portions  of  the  interface. 


A.2.5  Summary 

Many  of  the  methods  for  measuring  performance  that  have  been  discussed  above  have  been 
proposed  throughout  the  COMDAT  project  although  most  were  not  implemented  because  of 
logistical  or  practical  limitations  (see  section  9.2.2).  Thus,  the  lessons  learned  in  this  area  from  the 
manning  affordability  project  are  of  particular  importance  in  interpreting  the  recommendations  for 
H1L  measurement  of  performance  in  the  COMDAT  TDP. 
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Acronyms 


Acronym  or  Abbreviation 

Definitions 

2-D 

2-dimensional 

3-D 

3 -dimensional 

AMP 

Amplification  (as  in  ID  AMP,  PRI  AMP) 

ARRO 

Air  Raid  Reporting  Operator 

ASPO 

Anti  Submarine  Plotting  Operator 

ASWC 

Assistant  Sensor  Weapons  Controller 

C2 

Command  and  Control 

CCS 

Command  and  Control  System 

CDS  WG 

Command  Decision  Support  Working  Group 

CFNOS 

Canadian  Forces  Naval  Operations  School 

CSFAB 

Combat  Systems  Functional  Allocation  Board 

CO 

Commanding  Officer 

COA 

Course  Of  Action 

COMDAT 

Command  Decision  Aid  Technology 

CONOPS 

Concept  of  Operations 

CPF 

Canadian  Patrol  Frigate 

CSTC 

Combat  Systems  Training  Centre 

CTA 

Cognitive  Task  Analysis 

CWA 

Cognitive  Work  Analysis 

DL 

Decision  Ladder 

D-S 

Dempster-Shafer 

DRDC 

Defence  Research  and  Development  Canada 

ELINT 

Electronic  Intelligence 

ESM 

Electronic  Support  Measure 

EW 

Electronic  Warfare 

EWS 

Electronic  Warfare  Supervisor 

GCCS 

Global  Command  and  Control  System 

GDB 

Global  Database 
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Acronym  or  Abbreviation 

Definitions 

GUI 

Graphical  User  Interface 

HCI 

Human  Computer  Interaction 

HF 

Human  Factors 

HGA 

Hierarchical  Goal  Analysis 

HIL 

Human-In-the-Loop 

HMCCS 

Halifax  Class  Modernization  Command  and  Control  System 

HSI 

Human  Systems  Integration 

HUM 

Human  Resources  and  Performance  Group 

ICE 

Integrated  Command  Environment 

ID 

Identification 

IFF 

Identification  Friend  or  Foe 

IMD 

Information  Management  Director 

IP/PCT 

Information  Processing/Perceptual  Control  Theory 

IPME 

Integrated  Performance  Modelling  Environment 

MFTA 

Multi  Function  Task  Analysis 

MMWS 

Multi  Modal  Work  Station 

MOP 

Measure  of  Performance 

MS 

Microsoft 

MSDF 

Multi-Source  Data  Fusion 

NATO 

North  Atlantic  Treaty  Organization 

NCOT 

Naval  Combat  Operator  Trainer 

NTDS 

Naval  Tactical  Data  System 

OMI 

Operator  Machine  Interface 

OOW 

Officer  Of  the  Watch 

Ops  Room 

Operations  Room 

ORO 

Operations  Room  Officer 

ORS 

Operations  Room  Supervisor 

ORTT 

Operations  Room  Team  Trainer 

OSD 

Operational  Sequence  Diagram 

OSF 

Open  Software  Foundation 

OTT2 

Operations  Team  Training  2 
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Acronym  or  Abbreviation 

Definitions 

PRI 

Primary 

Prob 

Probable 

R&D 

Research  and  Development 

SA 

Situation  Awareness 

SWC 

Sensor  Weapons  Coordinator 

T&E 

Test  and  Evaluation 

TD 

Technology  Demonstrator 

TDP 

Technology  Demonstration  Project 

TG 

Task  Group 

TP 

Technical  Panel 

TS 

Track  Supervisor 

TTCP 

The  Technical  Cooperation  Programme 

U.S. 

United  States 

VCS 

Variable  Coded  Symbology 
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The  purpose  of  the  COMmand  Decision  Aiding  Technology  (COMDAT)  Technology  Demonstrator 
Project  (TDP)  was  to  research  and  demonstrate  multi-source  data  fusion  (MSDF)  technologies  and  carry 
out  human  factors  (HF)  studies  to  support  upgrades  to  the  Flalifax  Class  Command  and  Control  System 
(CCS),  in  the  area  of  battle  space  awareness.  Since  the  requisite  F1F  knowledge  to  support  this  TDP  did 
not  exist,  the  F1F  component  included  the  generation  of  the  founding  knowledge  necessary  for  decision 
aid  development  in  future  projects.  The  major  work  items  included 

•  the  conduct  of  cognitive  and  functional  task  analyses  to  define  the  information  requirements  of 
the  major  Operations  Room  positions  on  Flalifax  Class  ships; 

•  the  development  of  human-in-the-loop  measures  of  performance  in  order  to  evaluate  strategies 
for  conducting  performance  evaluations,  define  current  performance  in  the  Operations  Room, 
and  evaluate  technologies  including  MSDF; 

•  recommendations  for  improvements  to  human  computer  interfaces  in  the  existing  CCS  and  the 
Technology  Demonstrator  (TD); 

•  the  development  of  a  concept  of  operations  for  user  involvement  in  MSDF ;  and 

•  support  for  an  evaluation  of  operator-system  picture  compilation  performance  using  the  final 
TD. 

The  detailed  results  of  the  F1F  work  have  been  published  in  a  series  of  reports  over  the  course  of  the 
project  which  lasted  from  June  2000  to  March  2007.  This  final  report  provides  a  summary  of  the  reports 
along  with  related  material  that  informed  and  supplemented  this  work,  some  lessons  learned,  and 
recommendations  for  future  work.  Overall,  we  have  a  better  understanding  of  the  usefulness  of  the 
methods  that  were  investigated  as  well  as  considerably  more  knowledge  about  the  information 
requirements  and  processes  involved  in  building  the  tactical  picture  in  the  Flalifax  Class  Operations 
Room.  In  addition,  we  have  generated  improved  guidance  for  designing  the  interfaces  needed  to  support 
operator  goals  and  tasks  as  well  as  a  better  understanding  of  the  research  needed  to  build  even  more 
effective  interfaces  and  decision  support  systems  for  naval  command  and  control. 


Le  projet  de  demonstration  de  la  technologie  (PDT)  d'aide  aux  decisions  de  commandement  (COMDAT) 
avait  pour  objet  d’etudier  les  technologies  de  fusion  de  donnees  multi-sources  (MSDF)  et  d’en  faire  la 
demonstration  ainsi  que  de  mener  des  etudes  des  facteurs  humains  (FIT)  a  l’appui  de  mises  a  niveau  du 
systeme  de  commandement  et  de  controle  (C2)  des  navires  de  la  classe  Flalifax,  dans  le  domaine  de  la 
connaissance  de  Tespace  de  combat.  Comme  la  connaissance  des  FIT  necessaire  pour  appuyer  ce  PDT 
n’existait  pas,  la  composante  FIT  comprenait  la  generation  des  connaissances  de  base  necessaires  a 
Felaboration  d’aides  aux  decisions  dans  des  projets  futurs.  Les  principales  taches  de  recherche 
comprenaient : 

•  F  execution  d’analyses  des  taches  cognitives  et  fonctionnelles  pour  definir  les  exigences 
d’information  des  postes  principaux  dans  la  salle  des  operations  des  navires  de  la  classe  Flalifax; 

•  Felaboration  de  mesures  de  rendement  avec  un  chainon  humain  dans  la  boucle  afin  d’evaluer  des 
strategies  devaluation  du  rendement,  de  definir  le  rendement  actuel  dans  la  salle  des  operations 
et  d’evaluer  les  technologies,  y  compris  la  MSDF; 

•  des  recommandations  d’amelioration  des  interfaces  humain-ordinateur  dans  le  systeme  C2 _ 
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existant  et  le  demonstrates  de  la  technologie  (DT); 

•  T  elaboration  d’un  concept  des  operations  en  vue  de  la  contribution  des  utilisateurs  a  la  MSDF; 

•  le  soutien  d’une  evaluation  du  rendement  de  compilation  d’un  tableau  de  l’interface  operateur- 
systeme  au  moyen  du  DT  definitif. 

Les  resultats  detailles  des  travaux  en  FH  ont  ete  publies  dans  une  serie  de  rapports  tout  au  long  du  projet, 
qui  a  dure  de  juin  2000  a  mars  2007.  Le  present  rapport  final  resume  ces  rapports  de  meme  que  du 
materiel  connexe  qui  soutenait  et  completait  ces  travaux,  certaines  lemons  apprises  et  des 
recommandations  de  recherches  futures.  Somme  toute,  nous  avons  une  meilleure  comprehension  de 
l’utilite  des  methodes  etudiees,  ainsi  qu’une  connaissance  beaucoup  plus  complete  de  l’information 
requise  et  des  processus  necessaires  pour  constituer  le  tableau  tactique  dans  la  salle  des  operations  des 
navires  de  la  classe  Flalifax.  De  plus,  nous  avons  elabore  des  directives  ameliorees  pour  concevoir  les 
interfaces  necessaires  pour  appuyer  les  objectifs  et  taches  des  operateurs  et  acquis  une  meilleure 
comprehension  des  recherches  necessaires  pour  construire  des  interfaces  et  des  systemes  d’aide  aux 
decisions  encore  plus  efficaces  pour  le  commandement  et  le  controle  navals. 
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